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Abstract 

The  Air  Force  Research  Laboratory’s  Sensors  Directorate  has  crafted  a  long-term 
layered  sensing  project  and  seeks  a  method  by  which  to  compare  different  architectural 
representations.  This  research  provides  an  executable  methodology  for  quantitative 
architecture  comparisons  based  on  interoperability  characters  and  measurements.  The 
methodology  has  two  components  and  is  demonstrated  on  an  urban  operations  mission 
thread  scenario.  The  layered  sensing  scenario  requires  a  variety  of  sensor-equipped 
platforms  to  shift  orbits  while  supporting  a  mission  (e.g.  supply  convoy)  reacting  to 
unplanned  events  (e.g.  improvised  explosive  device).  The  first  component  is  a  discrete 
event  simulation  capturing  relevant  sensor,  platform,  and  mission  operations  and 
providing  measures  of  effectiveness  (MOEs)  and  measures  of  performance  (MOPs).  The 
second  component  is  an  application  of  a  general  purpose  interoperability  measurement 
technique  applied  to  a  specific  scenario  demonstrating  blue  on  blue  (collaborative) 
interoperability.  The  results  from  experimental  component  comparisons  show  that 
changes  in  interoperability  measurements  do  not  always  reflect  the  magnitude  of  changes 
in  mission  effectiveness  or  system  performance.  For  net-centric  applications,  changes  in 
the  number  of  process  paths  may  be  a  better  indicator  for  the  degree  of  interoperability 
present  in  a  given  architecture. 
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LINKING  INTEROPERABILITY  CHARACTERS  AND  MEASURES  OF 


EFFECTIVENESS:  A  METHODOLOGY  FOR  EVALUATING  ARCHITECTURES 


1.  Introduction 


1.1  Background 

The  conflicts  in  Iraq  and  Afghanistan  brought  new  challenges  to  an  American 
military  force  largely  designed  to  combat  other  conventional  forces  consisting  of  tanks, 
aircraft,  ships,  and  large  numbers  of  organized,  readily  identifiable  troops.  These 
challenges  are  highlighted  by  comparisons  between  the  first  and  second  Gulf  Wars. 
During  the  first  Gulf  War  in  1990  and  1991,  US/coalition  military  doctrine,  equipment, 
and  personnel  proved  far  superior  to  the  similarly  structured  but  ill-prepared  Iraqi  forces. 

The  current  Iraqi  conflict  has  proven  to  be  more  difficult  than  the  first  Gulf  War. 
New  challenges  include  fighting  against  insurgent  groups  that  resort  to  unconventional 
tactics  and  terrorist-like  activities.  The  US  and  her  allies  are  fighting  enemies  that  hide  in 
a  variety  of  difficult  settings,  including  masquerading  as  civilians.  Defeating  these 
unconventional  groups  of  fighters  requires  coalition  forces  to  quickly  adapt  tactics  and 
technologies.  One  series  of  adaptations  grew  into  the  concept  of  persistent  surveillance. 
Fueled  by  new  sensors,  platforms,  and  command  and  control  capabilities,  persistent 
surveillance  seeks  as  an  ultimate  goal  total  Area  of  Responsibility  (AOR)  coverage:  all 
locations,  24  hours  a  day,  7  days  a  week,  with  the  ability  to  spatially  zoom-in  or  out  with 
different  sensors  as  mission  needs  dictate.  (1:1-4) 
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Persistent  surveillance  requires  highly  interoperable  sensors  and  platforms 

operating  with  a  robust  command  and  control  backbone.  Working  in  support  of  this 

concept,  the  Air  Force  Research  Laboratory’s  (AFRL’s)  Sensors  Directorate  (RY)  crafted 

a  layered  sensing  architecture.  Figure  1  shows  an  Operational  View-1  (OV-1)  for  the 

architecture.  An  AFRL  white  paper  describes  layered  sensing  in  the  following  manner: 

“Layered  Sensing  provides  military  and  homeland  security  decision  makers  at  all 
levels  with  timely,  actionable,  trusted,  and  relevant  information  necessary  for 
situational  awareness  to  ensure  their  decisions  achieve  the  desired 
military  /humanitarian  effects.  Layered  Sensing  is  characterized  by  the 
appropriate  sensor  or  combination  of  sensors/platforms,  infrastructure  and 
exploitation  capabilities  to  generate  that  situation  awareness  and  directly  support 
delivery  of  “tailored  effects”.”  (l:ii) 

Table  1  captures  important  characteristics  for  potential  layered  sensing  platforms  and 
sensors  which  are  shown  in  Figure  2  through  Figure  5. 


Figure  1.  Layered  Sensing  OV-1 
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Table  1.  Layered  Sensing  Systems 


Platform 

Sensor 

Description 

Lair 

E/O 

Twin  prop  engine,  fixed  wing  aircraft,  E/O  sensor  with 
4-km  radius  Field  of  View  (FOV),  and  7-hr  mission  time 

Argus-IS 

E/O 

Single  engine,  rotary  wing  unmanned  aerial  vehicle,  E/O 
sensor  with  3-km  radius  FOV,  and  20-hr  mission  time 

Gotcha 

SAR 

ISR  pallet  aboard  fixed  wing  cargo  aircraft  (i.e.  C-130), 
SAR  sensor  with  20-km  radius  FOV,  6-hr  mission  time 

Nite  stare 

IR 

Twin  prop  engine,  fixed  wing  aircraft,  IR  sensor  with 
5-km  radius  FOV,  and  7-hr  mission  time 

Generic 

E/O,  IR,  SAR 

Twin  prop  engine,  fixed  wing  unmanned  aerial  vehicle, 
E/O,  IR,  and  SAR  sensors  with  10-km  radius  FOV,  and 
10-hr  mission  time 

AFRL/RY  seeks  a  method  by  which  to  compare  different  layered  sensing 
architectures  as  part  of  the  on-going  developmental  effort.  Useful  architecture 
comparisons  will  aid  acquisition  decision  making  as  layered  sensing  evolves  towards  an 
initial  fielding  in  the  2015  timeframe.  The  Sensors  Directorate  desire  for  a  quantitative 
comparison  methodology  provided  another  opportunity  for  Air  Force  Institute  of 
Technology  (AFIT)  participation  in  layered  sensing  research. 

The  AFIT  Department  of  Systems  and  Engineering  Management  (AFIT/ENV) 
recently  published  a  dissertation  that  hypothesized  a  novel  technique  for  measuring 
interoperability  across  a  broad  spectrum  of  systems  and  applications  (Ford,  2008).  The 
dissertation  sample  applications  focused  on  confrontational  interoperability  (blue  system 
interoperability  with  or  against  red  systems).  The  results  of  the  research  were  intriguing, 
and  AFIT/ENV  wished  to  further  test  the  technique  by  incorporating  scenarios  that 
measured  collaborative  interoperability  (blue  system  interoperability  with  other  blue 
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systems).  An  AFRL/RY  desire  to  evaluated  layered  sensing  architectures  coupled  nicely 


with  the  AFIT/ENV  interest  in  additional  testing  of  Ford’s  technique. 


Figure  2.  Lair/Nitestare:  C-12  Huron  (Electro-optical  and  Infrared  Sensors) 


Joint  doctrine  provides  a  concise  definition  of  interoperability:  it  is  “the  ability  to 
operate  in  synergy  in  the  execution  of  assigned  tasks.”  (2:2)  The  definition  can  be 
expanded:  “the  condition  achieved  among  communications-electronics  systems  or  items 
of  communications-electronics  equipment  when  information  or  services  can  be 
exchanged  directly  and  satisfactorily  between  them  and/or  their  users.”  (2:2)  Obtaining 
a  general  definition  is  a  necessary  first  step;  however,  providing  specific  examples  of 
worthwhile  interoperability  and  quantitatively  measuring  impact  on  mission  effectiveness 
is  more  challenging. 
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1.2  Problem  Statement 


The  research  addresses  this  fundamental  problem:  can  changes  in  architecture  be 
quantitatively  linked  to  changes  in  mission  effectiveness?  The  research  focus  is 
narrowed  down  to  only  those  changes  in  architecture  which  relate  to  measurements  of 
interoperability.  The  primary  approach  for  solving  this  problem  links  changes  in 
interoperability  measurements  (elements  of  architectural  designs)  to  changes  in  measures 
of  effectiveness  (MOEs)  or  performance  (MOPs)  through  discrete  event  simulation. 

1.3  Research  Hypothesis/Questions/Objectives 

This  section  presents  the  graduate  research  project  hypothesis.  The  hypothesis  is 
fleshed  out  with  an  accompanying  set  of  questions  that  bound  the  research  and  objectives 
that  focus  work  on  specific  goals. 

1.3.1  Hypothesis 

AFIT  asserts  that  MOE  values  characterizing  the  performance  of  a  collection  of 
systems  will  vary  as  a  function  of  the  set  of  interoperability  characters  inherent  to  that 
collection  of  systems.  Further,  the  addition  of  new  interoperability  characters  will  affect 
the  MOE  values  that  objectively  describe  the  collection  of  systems.  Lastly,  the  effect  of  a 
new  interoperability  character  may  be  positive  or  negative,  depending  upon  its  influence 
on  the  collection  of  systems.  Said  another  way,  more  interoperability  may  not  always  be 
good. 
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1.3.2  Questions 


The  questions  that  bound  the  research  are: 

1.  Can  a  collection  of  systems  be  at  least  partially  characterized  by  their 
collaborative  interoperability  characters? 

2.  Can  the  interoperability  measurement  methodology  derived  by  Ford  in  2008  be 
executed  for  scenarios  demonstrating  collaborative  interoperability? 

3.  Will  the  interoperability  characterization  manifest  itself  in  a  set  of  MOEs,  the 
values  of  which  are  calculated  by  a  discrete  event  simulation? 

4.  Can  a  discrete  event  simulation  adequately  capture  the  significant  operational 
characteristics  of  a  scenario  drawn  from  a  mission  thread  of  interest  to  AFRL/RY? 

5.  Can  a  methodology  be  crafted  to  support  future  AFRL/RY  architectural 
decision  making? 


Figure  3.  Argus-IS:  A-160  Hummingbird  (Electro-optical  Sensor) 
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1.3.3  Objectives 

The  objectives  that  focused  the  research  are: 

1.  Execute  Ford’s  methodology  for  scenarios  incorporating  collaborative 
interoperability. 

2.  Identify  an  operational  mission  thread  and  a  specific  scenario  relevant  to 
AFRL/RY  interests. 

3.  Build  a  discrete  event  simulation  that  adequately  captures  relevant  mission 
thread  and  scenario  characteristics. 

4.  Identify  a  set  of  interoperability  characteristics  that  adequately  capture  current 
and  potential  future  layered  sensing  attributes. 

5.  Incorporate  appropriate  systems  engineering  principles  learned  throughout  the 
AFIT  Intermediate  Developmental  Education  (IDE)  program.  Develop  static  and 
dynamic  representations  of  layered  sensing  in  order  to  lay  the  foundation  for  a  discrete 
event  simulation. 

6.  Determine  a  set  of  MOEs  that  demonstrate  adequate  traceability  and  capture 
relevant  layered  sensing  performance. 

7.  Capture  values  for  these  MOEs  (done  with  the  discrete  event  simulation) 

8.  Demonstrate  a  link  between  interoperability  characters  and  MOEs. 

1.4  Methodology 

The  research  methodology  has  two  components  and  is  demonstrated  on  an  urban 
operations  mission  thread  scenario.  The  scenario  requires  electro-optical  (E/O),  infrared 
(IR),  and  synthetic  aperture  radar  (SAR)  sensor-equipped  platforms  to  shift  orbits  while 
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supporting  a  mission  (e.g.  supply  convoy)  reacting  to  an  unplanned  event  (e.g. 
improvised  explosive  device  planted  along  the  convoy  route  of  travel).  The  scenario  was 
selected  from  three  use  cases  developed  during  discussions  with  AFRL/RY.  The  selected 
scenario  is  a  case  of  high  interest  to  both  the  sponsor  and  the  researchers. 


Figure  4.  Gotcha:  ISR  Pallet  aboard  a  Cargo  Aircraft  (e.g.  C-130)  (SAR  Sensor) 

The  first  methodology  component  captures  relevant  sensor,  platform,  and  mission 
operations.  The  primary  simulation  outputs  are  MOE  and  MOP  values.  The  MOEs  and 
MOPs  are  drawn  from  the  AFRL  layered  sensing  attributes.  The  foundation  of  the 
discrete  event  simulation  rests  on  static  (object  diagram)  and  dynamic  (sequence 
diagram)  representations  of  the  selected  use  case.  An  operational  activity  model  was  also 
developed  to  seed  the  simulation  effort. 
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The  second  component  is  the  interoperability  measurement  technique  developed 
by  Ford  in  2008.  In  this  case,  the  research  attempts  to  link  collaborative  interoperability 
measures  to  MOEs.  The  measurement  approach  identifies  interoperability  characters  in 
the  layered  sensing  architecture,  links  the  various  characters  in  a  matrix  calculation,  and 
calculates  the  so-called  measure  of  interoperability.  Different  layered  sensing 
architectures  will  have  different  sets  of  interoperability  characters.  As  such,  the 
technique  will  produce  a  different  interoperability  measure  for  each  sensing  architecture. 

Experimentation  shows  the  two  methodology  component  results  may  indeed  be 
linked.  When  the  linkage  is  present,  it  occurs  between  the  discrete  event  simulation- 
calculated  MOE  and  MOP  values  and  the  interoperability  measurement  made  by  Ford’s 
technique.  Further,  as  interoperability  characters  are  added  to  both  the  interoperability 
measurement  matrix  and  the  discrete  event  simulation,  the  MOE  and  MOP  values  change 
in  ways  that  do  not  always  correlate  well  with  interoperability  measurements.  More  to 
the  point,  the  addition  of  interoperability  characters  that  show  small  increases  in 
interoperability  measurements  may  cause  substantial  changes  in  MOE  and  MOP  values. 
In  other  cases,  there  is  no  quantifiable  linkage  between  interoperability  measurement  and 
MOE  or  MOP  value.  Although  not  conclusive,  the  set  of  numerical  experiments 
examined  during  this  research  does  further  define  the  relationship  between  changes  in 
interoperability  and  measures  of  mission  effectiveness. 
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1.5  Assumptions/Limitations 

There  are  a  number  of  assumptions  associated  with  the  research  approach  that 
bound  the  conclusions  and  implications  reached  during  the  project.  The  most  important 
assumptions  and  their  accompanying  limitations  are  summarized  in  this  section. 

The  concept  of  interoperability  measurement  is  both  diverse  and  complex.  As 
such,  a  series  of  important  assumptions  were  made  to  scope  the  research  down  to  a 
manageable,  yet  still  useful  and  interesting,  level  of  effort.  The  first  scoping  decision 
was  selecting  an  urban  mission  thread  instead  of  a  rural  mission  thread,  the  assumption 
being  that  urban  mission  threads  were  both  more  challenging  for  layered  sensing 
applications  and  more  interesting  to  the  research  sponsor. 

Once  an  urban  mission  thread  was  selected  three  specific  use  cases  were 
developed  in  conjunction  with  the  AFRL  sponsor.  One  of  these  use  cases  centered  on  a 
blue  force  reaction  to  an  unplanned  event.  The  reaction  to  an  unplanned  event  was 
selected  because  the  researchers  assumed  it  was  both  the  most  interesting  and  the  most 
stressing  of  the  three  scenarios  for  current  layered  sensing  architectures. 

There  are  limitations  associated  with  the  scoping  effort.  The  most  significant  is 
that  selection  of  an  urban  mission  thread  removed  hyperspectral  sensors  from  inclusion  in 
the  discrete  event  simulation.  Although  worth  noting,  this  limitation  has  no  impact  on  the 
conclusions  reached  after  examining  the  numerical  experiments. 

More  assumptions  were  made  regarding  the  types  of  sensors  and  platforms 
included  in  the  discrete  event  simulation.  Specifically,  three  sensor  types  were  selected: 
E/O,  IR,  and  SAR.  Although  layered  sensing  could  ultimately  incorporate  a  large 
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number  of  different  sensor  types,  the  three  selected  provided  enough  flexibility  to  deal 
with  challenges  presented  during  simulated  missions  (e.g.  day  versus  night,  good  weather 
versus  bad). 


Figure  5.  Notional  MQ-X  Reaper- like  Unmanned  Arial  Vehicle  (Multiple  Sensors) 

The  number  of  platform  types  was  capped  at  five.  There  is  no  cap  for  total 
number  of  platforms,  just  for  different  types.  Five  platform  types  coupled  to  three  sensor 
options  demonstrate  that  the  quantitative  architecture  comparison  methodology  is  viable 
even  if  every  possible  future  manifestation  of  layered  sensing  cannot  be  mimicked  in  the 
current  simulation.  These  sensor  and  platform  assumptions  were  made  specifically  to 
simplify  the  discrete  event  simulation. 

A  spatial  construct  was  needed  for  the  simulation.  To  address  this  need,  the 
researchers  used  a  grid  composed  of  keypads  (individual  squares  in  the  grid)  to  represent 
the  mission  space.  Both  keypad  and  grid  size  are  inputs  provided  by  the  simulation  user. 
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Although  there  are  no  upper  or  lower  bounds  programmed  into  the  simulation,  the 
numerical  experiments  were  run  with  a  keypad  size  of  one  square  kilometer  and  a  grid 
that  measured  50  by  50  keypads  (50  by  50  kilometers).  A  spatial  construct  of  this  nature 
corresponds  well  to  the  actual  size  of  an  urban  area  of  interest,  is  large  enough  to 
incorporate  various  sensor  footprints,  allows  for  a  range  of  distances  between  sensors  and 
the  location  of  interest,  and  is  appropriately  scaled  for  ranges  of  typical  ground  missions. 
Even  though  the  space  over  which  the  simulated  sensors  and  platforms  operate  is  limited, 
the  limitation  does  not  detract  from  either  the  numerical  experimentation  or  the  results 
derived  from  it. 

The  simulation  starts  with  the  unplanned  event  to  which  the  mission  must 
respond.  Once  started  the  simulation  runs  for  24  hours  and  the  run  time  does  not  change 
from  one  run  to  the  next.  Most  missions  of  the  type  mimicked  in  the  simulation  would 
not  last  24  hours.  This  departure  from  reality  is  tolerated  because  the  simulation  length 
was  selected  to  stress  layered  sensing  architectures.  Specifically,  the  24-hour  mission 
length  greatly  exceeds  loiter  time  for  the  platforms  in  the  AOR  at  the  time  the  unplanned 
event  occurred.  This  timing  epoch  all  but  forces  layered  sensing  architectures  to 
reconstitute  platforms  and  sensors  in  order  to  cover  the  entire  mission.  Further, 
transitions  between  day  and  night,  which  force  architectures  to  blend  sensor  types  or 
suffer  in  the  MOE-based  performance  measurement,  have  been  modeled  as  well. 

The  simulation  has  a  key  decision  point  early  in  each  run:  select  sensor  and 
platform  combinations  in  the  AOR  at  the  time  of  the  unplanned  event  to  respond  to  the 
event  (shift  orbits  to  ensure  the  event  location  falls  within  the  sensor  footprint).  After  this 
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decision  is  made,  any  new  platform  tasked  to  cover  the  mission  comes  from  its  respective 
home  airfield.  In  other  words,  platforms  in  the  AOR  not  initially  selected  to  cover  the 
event  are  not  later  tasked  to  backfill  platforms  as  loiter  times  expire.  This  condition 
would  not  hold  in  most  real  contingency  scenarios.  In  the  simulation,  it  is  assumed  that 
these  platform/sensor  combinations  are  allocated  to  other  missions  and  cannot  be 
retasked  after  the  initial  tasking  decision  is  made.  This  limitation  was  adopted  because  it 
is  more  stressing  for  layered  sensing  architectures.  It  requires  platforms  to  travel  farther 
distances  or  boast  longer  loiter  times.  The  assumption  also  implies  a  heavier  than  normal 
tasking  load. 

The  simulation  has  a  user  interface.  The  interface  is  an  Excel  spreadsheet  that 
requests  input  for  a  number  of  variables  needed  to  run  the  simulation.  For  example,  the 
user  can  select  both  the  total  number  of  platforms  and  platform  types  (as  long  as  types  do 
not  exceed  five).  The  user  can  also  select  certain  platform  performance  factors  like 
speed.  Others,  like  altitude,  are  not  requested.  Performance  characteristics  that  play  a 
role  in  architecture  evaluation  are  requested  from  the  user;  those  that  do  not  influence  the 
evaluation  or  are  of  no  interest  to  the  sponsor  are  not  requested.  So,  although  altitude 
impacts  sensor  ability  to  resolve  targets,  AFRL/RY  assumes  all  platforms  will  be 
operating  nominally  at  20,000  ft  MSF  and  sees  no  need  to  work  altitude  into  the 
methodology. 

All  of  the  previously  discussed  simulation  assumptions  and  limitations  drive 
home  one  key  point:  the  simulation  is  not  reality.  Creating  a  simulation  that  captures  all 
aspects  of  the  modeled  use  case  accurately  is  well  beyond  the  scope  of  this  research.  The 
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motivation  for  creating  this  simulation  was  modeling  performance  in  a  manner  that 
allows  for  consistent  MOE  measurement  while  interoperability  characters  are  varied 
during  different  experimental  runs.  The  simulation  is  deemed  a  success  as  far  as  this 
limited  goal  is  concerned. 

There  are  many  potential  sources  for  interoperability  characters  relevant  to  the 
problem  at  hand.  In  order  to  properly  constrain  character  selection,  the  researchers 
identified  two  primary  motivations:  relevancy  to  layered  sensing  and  an  appropriate  level 
of  Department  of  Defense  (DoD)  vetting.  To  accomplish  both  of  these  ends,  characters 
from  three  Joint  Capability  Areas  (JCAs)  were  used  for  experimentation:  Battlespace 
Awareness,  Command  and  Control,  and  Net-Centric.  These  three  sources  provided 
ample  characters  to  use  for  experimentation  and  adequately  addressed  hypothesis 
requirements. 

Lastly,  the  work  done  in  this  graduate  research  project  retains  all  of  the  strengths 
and  weaknesses,  both  known  and  unknown,  of  Ford’s  interoperability  measurement 
methodology.  No  attempts  were  made  to  improve  his  technique,  only  to  apply  it  to  a 
specific  collaborative  interoperability  scenario. 

1.6  Implications 

The  research  produced  results  supporting  the  following  implications: 

1.  Ford’s  interoperability  measurement  technique  can,  in  some  cases,  be  used  to 
relate  collaborative  interoperability  to  mission  effectiveness  measures. 

2.  Discrete  event  simulations  can  adequately  capture  the  performance  of  a 
collection  of  systems  and  characterize  this  performance  by  capturing  MOEs  and  MOPs. 
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3.  Changing  the  interoperability  characters  present  in  a  given  collection  of 
systems  can  be  quantified  by  changes  in  interoperability  measurements.  Further, 
interoperability  character  changes  can  sometimes  be  linked  to  changes  in  MOE  and  MOP 
values. 

4.  Identifying  interoperability  characters  in  a  given  architecture  and  capturing 
resultant  interoperability  measurements  (with  Ford’s  technique  and  MOE  values)  allows 
for  improving  architecture  designs  by  changing  interoperability  characters.  Further, 
ranked  comparisons  can  be  made  between  different  architectures  performing  the  same 
mission. 

The  last  implication  lays  the  foundation  for  a  quantitative  architecture  comparison 
methodology  that  can  lend  analytical  rigor  to  future  layered  sensing  acquisition  actions 
undertaken  by  AFRL/RY. 

1.7  Document  Outline 

This  graduate  research  project  report  now  proceeds  to  Chapter  2,  which  is  a 
review  of  literature  relevant  to  the  work  conducted  over  the  last  three  academic  quarters. 
Five  pieces  of  literature  (or  at  least  literature-like  information)  were  particularly 
important.  Chapter  3  provides  an  in-depth  treatment  of  the  methodology  used  to  conduct 
the  numerical  experiments  and  make  the  interoperability  measurements.  Chapter  4 
presents  the  results  of  the  experimentation  and  includes  additional  analysis  done  to 
properly  scope  the  findings.  The  last  chapter,  Chapter  5,  contains  the  conclusion  and 
includes  recommendations  for  further  research. 
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2.  Literature  Review 


2.1  Overview 

Interoperability  is  a  broad  topic  of  interest  to  a  number  of  diverse  communities. 

As  such,  there  is  a  substantial  offering  of  literature  describing  different  aspects  of 
interoperability.  It  is  impossible  to  capture  all  of  the  writing  worthy  of  review  in  a  single 
chapter.  Instead,  a  summary  of  the  most  influential  documents  is  provided.  Each  section 
briefly  describes  one  source  and  explains  its  relevancy  to  the  research. 

2.2  Quantification  of  Interoperability  -  Early  Thoughts 

In  May  of  1989  Mensh,  Kite,  and  Darby  published  an  article  in  the  Naval 
Engineers  Journal  that  linked  quantitative  interoperability  analyses  to  MOEs  and  MOPs. 
The  analytical  technique  used  to  link  interoperability  to  MOEs  and  MOPs  rested  on  a 
depth  of  experience  garnered  from  years  spent  working  naval  interoperability  issues.  The 
methodology  presented  in  the  paper  was  applied  to  an  August  1987  Tactical  Information 
Management  Exercise.  Although  “elementary  and  based  on  fundamental  principles,” 
(7:251)  the  concept  developed  by  these  three  researchers  seeded  several  key  ideas  for  the 
AFIT  students. 

First,  defining  “interoperability  component”  expanded  the  1989  Joint  Chiefs  of 
Staff  definition  of  interoperability  in  a  manner  supporting  quantitative  measurement  and 
so  enabled  a  precursor  technique  to  the  AFIT  methodology  developed  by  Ford  in  2008. 
The  concept  of  an  interoperability  component  also  likely  influenced  the  AFIT  definition 
of  “interoperability  character.” 
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Second,  Mensh  et  al  linked  quantification  of  interoperability  to  MOPs  and,  more 
important  for  the  AFIT  research,  MOEs.  This  linkage  is  the  linchpin  upon  which  the 
AFIT  interoperability  comparisons  are  based. 

Third,  the  Navy  researchers  calculated  MOE  values  by  using  numerical 
simulations  —  an  approach  emulated  by  AFIT. 

Finally,  Mensh  and  his  partners  used  “basic  truth  table  theory  in  conjunction  with 
logic  equations  to  evaluate  the  interoperability  components.”  (7:251)  This  Boolean 
representation  of  interoperability  (the  component  is  either  completely  present  or 
completely  absent)  is  indeed  basic.  It  is  also  appealing,  elegant,  and  justifiable  for  some 
applications.  The  current  AFIT  research  uses  the  same  type  of  representation  for  the 
presence  of  interoperability  characters  and  their  impact  to  layered  sensing  architectures. 

2.3  Layered  Sensing 

In  2008  AFRF/RY  published  a  white  paper  called  “Fayered  Sensing:  Its 
Definition,  Attributes,  and  Guiding  Principles  for  AFRF  Strategic  Technology 
Development.”  This  paper  described  the  need  for  a  layered  sensing  architecture  due  to 
the  changing  nature  of  irregular  warfare  and  the  current  demand  for  Intelligence, 
Surveillance  and  Reconnaissance  (ISR).  To  accomplish  this  mission,  AFRF  plans  to 
develop  technologies  that  will  “certainly  include  various  sensor  systems,  including  audio, 
visual,  chemical,  biological,  radio  frequency,  microwave,  x-ray,  infrared,  ultraviolet,  [...] 
acoustic,  and  so  on.  Fayered  Sensing  is  characterized  by  the  appropriate  sensor  or 
combination  of  sensors/platforms,  infrastructure  and  exploitation  capabilities.”  (1:11) 
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Clearly,  the  AFRL  white  paper  is  important  because  it  describes  what  layered 
sensing  is,  at  least  as  far  as  the  Sensors  Directorate  is  concerned.  The  paper, 
supplemented  by  conversations  with  the  sponsor  and  other  AFRL  provided 
documentation,  helped  scope  the  research  effort  while  keeping  the  project  relevant  to  the 
Sensors  Directorate  mission. 

For  AFIT,  one  of  the  most  important  aspects  of  the  AFRL  white  paper  is  the 
attributes  that  characterize  layered  sensing.  These  attributes  are  shown  in  Table  2. 
Pulling  attributes  from  the  AFRL  paper  that  defines  layered  sensing  provides  traceability 
and  ensures  the  selected  attributes  are  well  grounded  in  the  customer’s  value  system. 


Table  2.  Layered  Sensing  Attributes 


Persistent  Coverage* 

Wide  Area  Coverage* 

Assured  Global  Access 

Engagement  Quality  Information 

Timeliness* 

Trusted  Sensing 

Information  Triage 

Robust,  Agile,  Adaptable* 

Spectrum  Dominance  and  Control* 

Anticipatory  Observations  and  Interactive  Engagements 

Tailored  Performance 

Affordable  Open  System  Architecture 

The  MOEs  used  to  gauge  layered  sensing  performance  must  be  linked  to 
attributes.  By  providing  validated  attributes  in  their  white  paper,  AFRL  certainly 
facilitated  AFIT  efforts  to  derive  MOEs  relevant  to  layered  sensing.  The  attributes 
marked  with  an  asterisk  in  Table  2  are  the  attributes  to  which  AFIT  has  linked  MOEs. 
Table  3  shows  the  MOEs  calculated  during  the  simulation  and  the  attributes  to  which 
they  are  linked. 
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Table  3.  Attributes  and  Measures  of  Effectiveness 


Attribute 

Measure  of  Effectiveness 

Persistent  Coverage 

-  MOE  1 :  Percentage  of  time  mission  is 
covered  by  one  sensor 

Wide  Area  Coverage 

-  MOE  2:  Percentage  of  the  Area  of 
Responsibility  covered  by  sensors 

Timeliness 

-  MOE  3:  Time  for  information  to  pass 
from  sensor  to  ground  forces  (fully 
processed  upon  arrival) 

Robust,  Agile,  Adaptable 

-  MOE  4:  Layered  sensing  mission  failures 
in  a  24-hour  period 

-  MOE  5:  Average  time  taken  to  begin 
coverage  after  the  unplanned  event  occurs 

Spectrum  Dominance  and  Control 

-  MOE  6:  Percentage  of  time  mission 
covered  by  at  least  two  platforms 

2.4  Measuring  Interoperability 

The  authors  used  the  aforementioned  AFIT  technique  created  by  Ford  in  2008  to 
measure  interoperability.  These  measurements  are  one  of  the  two  sets  of  metrics  used  to 
evaluate  and  compare  layered  sensing  architectures,  so  understanding  the  technique  was 
an  important  part  of  the  research  effort.  Also,  one  of  the  research  goals  was  to 
demonstrate  the  feasibility  of  extending  Ford’s  method  to  use  collaborative 
interoperability  measurements.  Ford’s  dissertation  provided  the  needed  understanding 
and  was  a  key  reference  throughout  the  project. 

A  short  summary  of  Ford’s  technique  shows  several  steps  must  be  taken  in  order 
to  produce  a  measurement  value.  First,  the  systems  of  interest  must  be  identified.  Next, 
an  accompanying  set  of  interoperability  characters  must  be  selected.  The  systems  then 
need  to  be  instantiated  whereby  a  matrix  (mating  systems  to  characters)  is  populated  with 
a  Boolean  representation  of  system  capabilities  based  on  which  interoperability 
characters  are  present  (although  not  used  in  this  research  project,  Ford’s  technique  can 
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also  represent  interoperability  characters  with  real  numbers).  Once  these  steps  have  been 
accomplished,  a  measure  of  interoperability  can  be  calculated.  This  is  a  simple 
application  of  Ford’s  method,  but  appropriate  for  the  task  at  hand. 

The  derivation  of  the  technique  is  involved,  and  will  not  be  repeated  here. 
Interested  readers  are  directed  to  reference  6  (especially  the  third  chapter)  in  the 
bibliography  for  a  detailed  treatment  of  the  technique.  Measurement  calculations  made 
during  specific  experiments  are  discussed  in  detail  in  Chapters  3  and  4  and  Appendix  A3. 

2.5  Interoperability  Characters 

As  stated  earlier,  a  set  of  relevant  interoperability  characters  must  be  identified  in 
order  to  successfully  execute  Ford’s  methodology.  These  characters  need  to  reflect  some 
aspect  of  interoperability,  must  be  either  present  in  the  modeled  use  case  or  capable  of 
being  added  to  it,  and  they  should  demonstrate  traceability  back  to  a  vetted  DoD  source. 

Three  JCAs  were  identified  that  meet  all  of  the  above  criteria.  These  JCAs  are 
Battlespace  Awareness,  Command  and  Control,  and  Net-centric.  The  J7  under  the 
Chairman  of  the  Joint  Chiefs  of  Staff  maintains  a  list  of  JCA-related  definitions  on  the 
Defense  Technical  Information  Center’s  website  (5).  The  specific  interoperability 
characters  used  during  research  were  drawn  from  this  list  and  are  shown  in  Table  4. 
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Table  4.  Interoperability  Characters 


JCA  Definition 
Number 

Character 

2 

Battlespace  Awareness 

2.1 

Intelligence,  Surveillance,  and  Reconnaissance 

2.1.2 

Collection 

2. 1.2. 3 

Imagery  Collection 

2. 1.2. 3.1 

Electro-Optical  Imagery  Collection 

2.1. 2.3. 1.1 

Panchromatic  Imagery  Collection 

2.1. 2.3. 1.2 

Infrared  Imagery  Collection 

2.1.2.3.2 

RADAR  Imagery  Collection 

2.1.3 

Processing/Exploitation 

2.1.3. 1 

Data  Transformation 

2. 1.3. 2 

Objective/Target  Categorization 

2.1.4 

Analysis  and  Production 

2.1.5 

Intelligence,  Surveillance  and  Reconnaissance  Dissemination 

5 

Command  and  Control 

5.5 

Direct 

5.5.2 

Task 

5.5.2. 1 

Synchronize  Operations 

5.5. 2.2 

Issue  Plans 

5.5. 2.3 

Issue  Orders 

6 

Net-centric 

6.1 

Information  Transport  (IT) 

6.1.2 

Wireless  Transmission 

6. 1.2.1 

Line  of  Sight 

6. 1.2. 2 

Beyond  Line  of  Sight 

2.6  A  Process  for  Providing  Intelligence,  Surveillance,  and  Reconnaissance  Support 

Joint  Publication  2-01  (JP  2-01),  Joint  and  National  Intelligence  Support  to 
Military  Operations,  describes  the  process  used  to  provide  finished  intelligence  to  US 
military  forces.  This  doctrinal  process  description  is  important  to  the  AFIT  research 
because  it  provided  a  guide  for  constructing  the  simulation  component  that  models  how 
intelligence  data  passes  from  sensors  through  the  Joint  or  Combined  Air  Operations 
Center  (JAOC  or  CAOC)  to  the  forces  that  require  support.  The  interoperability-related 
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MOE  used  to  gauge  system  performance  is  made  in  this  part  of  the  simulation,  so  a  DoD- 
sanctioned  process  was  an  essential  requirement  for  building  a  vetted  simulation. 

The  JP  2-01  process  has  the  following  components: 

1 .  Planning  and  direction 

2.  Collection 

3.  Processing  and  exploitation 

4.  Analysis  and  Production 

5.  Dissemination 

6.  Evaluation  and  feedback 

The  discrete  event  simulation  incorporates  components  two  through  five.  Both  planning 
and  direction  and  evaluation  and  feedback  are  beyond  the  scope  of  the  current  research. 

In  the  use  case  simulated  for  this  research,  collection  functions  are  performed 
primarily  by  the  sensor  and  its  accompanying  platform.  These  functions  potentially 
include  target  identification,  coordination  with  other  friendly  forces,  and  prioritization  of 
missions  and  resources. 

Processing  and  exploitation  may  occur  in  the  CAOC.  So  may  analysis  and 
production.  The  platforms  that  carry  the  sensors  may  also  carry  out  these  tasks. 

Functions  performed  by  these  components  encapsulate  the  process  of  transforming  raw 
data  into  finished  intelligence.  This  includes  merging  data  from  multiple  sources, 
integrating  data  with  different  formats,  interpretation,  analysis,  coordination,  and  crafting 
the  finished  product  into  a  medium  that  efficiently  delivers  the  intelligence  assessment. 

In  the  AFIT  simulation  dissemination  occurs  between  the  CAOC  and  the  ground 
forces  conducting  the  mission.  It  can  also  take  place  directly  between  the  sensors  and 
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ground  forces.  Both  the  method  of  dissemination  and  the  path  over  which  dissemination 
occurs  can  significantly  change  layered  sensing  performance,  so  numerical  experiments 
characterizing  dissemination  are  critically  important  to  the  AFIT  effort.  Dissemination  is 
not  just  transmission;  it  is  transmission  that  effectively  and  efficiently  couples 
intelligence  assessments  to  the  force  command  structure  that  needs  them. 

2.7  Arena  Discrete  Event  Simulations 

The  Arena  simulation  software  environment  produced  by  Rockwell  Software  Inc. 
was  the  tool  used  to  build  the  discrete  event  simulation.  The  literature  review  included 
searches  for  training  material  in  order  to  maximize  efficient  use  of  the  Arena  software. 
Although  perhaps  not  as  interesting  as  some  of  the  other  items  described  in  this  chapter, 
two  sources  of  Arena  knowledge  were  just  as  pivotal  for  project  success  as  any  other 
piece  of  literature  used  during  the  research. 

The  first  Arena  source  was  the  Arena  Basic  User’s  Guide  written  by  Rockwell.  It 
provided  the  elementary  descriptions  of  Arena  components,  programming  philosophy, 
and  syntax  necessary  to  get  the  simulation  effort  started.  (8) 

As  the  simulation  quickly  grew  in  complexity  the  SMARTs  library  (part  of  the 
robust  help  function  in  the  Arena  software  package)  became  the  most  important  source 
for  Arena  knowledge.  The  library  contained  a  large  number  of  SMART  files  that  were 
samples  of  Arena  code  simulating  processes  of  interest  for  capturing  discrete  events. 
These  smart  file  examples  helped  the  researchers  build  some  of  the  processes  included  in 
the  layered  sensing  simulation.  (9) 
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3.  Methodology 


3.1  Overview 

This  chapter  describes  the  methodology  used  to  look  for  a  correlation  between  the 
measurement  of  interoperability  and  MOEs  that  characterize  system  performance.  The 
description  begins  by  discussing  steps  taken  to  scope  the  problem  and  prepare  the 
simulation  framework.  Once  this  is  complete,  the  experimental  construct  is  discussed. 
Next,  the  calculation  of  the  interoperability  measurement  technique  is  explained. 
Execution  of  the  technique  is  linked  to  the  discrete  event  simulation;  each  experiment 
must  have  a  measure  of  interoperability  accompanied  by  a  set  of  MOEs  so  that 
comparisons  can  be  made  with  the  baseline  architecture.  As  such,  the  simulation 
approach  is  addressed  next.  The  chapter  culminates  with  a  discussion  of  how  the  two 
parts  of  the  experimental  process  -  interoperability  measurement  and  MOE  evaluation  - 
are  linked  to  look  for  the  correlation.  Lastly,  sample  test  plans  are  presented  to  facilitate 
the  presentation  of  results  in  Chapter  4. 

3.2  Scoping  and  Preparation 

The  layered  sensing  architecture  built  by  AFRL/RY  describes  a  system  of  systems 
used  to  perform  ISR  missions.  The  collection  of  systems  not  only  includes  platforms  and 
sensors,  but  functions  residing  in  the  CAOC  and  the  forces  requiring  ISR  support,  as 
well.  The  complex  nature  of  layered  sensing  made  system  familiarization  an  early  goal. 
Familiarization  also  facilitated  the  scoping  effort.  Most  of  the  system  information  came 
from  the  aforementioned  AFRL  white  paper,  discussions  with  the  research  sponsor, 
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various  briefings  and  documents,  and  presentations  given  during  an  AFRL  industry  day 
held  in  December  2008. 

Figure  6  shows  the  scoping  process  that  moved  the  research  effort  from  AFRL 
white  paper  to  experimental  construct.  There  were  three  separate  paths  of  study.  The 
first  path  identified  a  scenario  of  interest  which  fostered  development  of  the  discrete 
event  simulation.  The  second  path  identified  important  interoperability  characters.  The 
third  and  final  path  led  to  selection  of  the  MOEs.  Blocks  with  blue  text  denote  scoping 
steps  taken  by  AFIT.  Blocks  with  green  text  denote  AFRL  provided  information. 


Figure  6.  The  Scoping  Process 

Discussions  with  the  sponsor  quickly  focused  interest  on  an  urban  mission  thread. 
That  focus  drove  the  selection  of  platforms  and  sensors  discussed  in  Chapter  1,  all  of 
which  were  used  for  both  the  interoperability  measurement  and  the  MOE  evaluation 
produced  by  the  simulation.  Both  the  simulation  and  the  interoperability  measurement 
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technique  can  be  executed  for  other  platforms  or  sensors,  but  those  in  Table  1  (Chapter  1) 
best  captured  the  interests  of  the  sponsor  and  the  researchers. 

Once  the  mission  thread  and  accompanying  platforms  and  systems  were  selected, 
the  next  step  was  learning  how  layered  sensing  operates.  The  tool  of  choice  was  use  case 
development  done  in  conjunction  with  the  sponsor.  Three  use  cases  were  built  to 
describe  three  different  layered  sensing  applications  (see  Table  5  and  Appendix  1).  Of 
these  three,  UC  2  was  selected  for  study  because  it  presented  interesting  challenges  for 
any  layered  sensing  architecture,  and  so  constituted  a  useful  benchmark  for  comparison. 


Table  5.  Use  Case  Descriptions 


Use  Case 

Condition  Under  Test 

UC  1 :  Maintain  surveillance 
over  a  fixed  point 

Ability  to  identify  and  analyze  patterns  of  activity. 

Goal  is  evaluating  ability  to  successfully  recognize 
event  tip-offs 

UC  2:  Track  friendly  force 
movements  and  reactions 

Ability  to  react  to  changes  in  operational  conditions, 
especially  convoy  re-routing  in  response  to  enemy 
action.  Goal  is  evaluating  ability  to  maintain  platform 
and  sensor  support  to  ground  forces 

UC  3:  Survey  fixed  point  for 
forensic  analysis 

Ability  to  route  information  to  where  it  is  needed. 

Goal  is  evaluating  CAOC  ability  to  identify,  store, 
retrieve,  analyze,  and  disseminate  relevant  intelligence 

Selection  of  a  specific  scenario  (UC  2)  was  the  first  step  needed  for  construction 
of  the  discrete  event  simulation.  This  was  followed  by  creating  static  and  dynamic 
representations  of  the  layered  sensing  systems.  The  first  of  these  representations  was  an 
object  diagram  (see  Figure  7).  Although  static,  the  object  diagram  showed  what  objects 
needed  to  interact  in  the  simulation  in  order  to  accomplish  the  scenario  mission.  Insight 
was  also  gained  for  required  multiplicities  and  the  interactions  between  specific  objects. 
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Figure  7.  Layered  Sensing  Object  Diagram 
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The  next  model  built  was  a  sequence  diagram  (see  Figure  8).  This  diagram  was 
the  first  dynamic  view  of  the  system  and  provided  insight  into  what  operations  are  carried 
out  by  the  different  objects  and  the  order  (in  time)  in  which  they  must  occur. 

Establishing  the  sequence  of  actions  drove  construction  of  the  discrete  event  simulation 
and  also  assisted  in  MOE  selection. 

The  last  representation  built  by  the  researchers  was  the  activity  model  (see 
Figure  9).  This  model  explored  both  the  logical  entities  in  layered  sensing  and  the  data 
that  is  used  or  transferred  by  each  entity.  The  activity  model  also  helped  directly  transfer 
characteristics  captured  by  all  three  models  into  the  Arena  discrete  event  simulation 
software  environment. 

The  use  of  Unified  Modeling  Language  (UML)  tools  during  the  early  stages  of 
research  solidified  the  perception  of  layered  sensing,  its  system  components,  and  its 
mission  applications.  The  UML  models  were  key  enablers  for  crafting  a  robust  discrete 
event  simulation  that  met  the  needs  of  AFRL  and  AFIT. 
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Figure  8.  Layered  Sensing  Sequence  Diagram 
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Figure  9.  Layered  Sensing  Activity  Model 
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3.3  Experimental  Construct 


Ford  identified  a  process,  depicted  in  Figure  10,  for  measuring  interoperability. 
Although  Ford’s  application  was  interoperability  analysis,  the  current  researchers  use  this 
process  as  a  construct  for  both  components  of  the  experimental  procedure:  measuring 
interoperability  and  calculating  MOEs.  In  essence,  Figure  10  serves  as  a  map  that  guides 
the  interested  reader  through  the  experimental  setup. 


State 

Purpose 

Take 

Action 


Identify 

Process 


Perform 

Analysis 


Identify 

MoOEs 


Measure 

Interoper¬ 

ability 


Identify 

Systems 


Instantiate  Identify 

Systems  Characters 

V  _  V. 


Figure  10.  Ford’s  Interoperability  Measurement  Process  (6:28) 

By  using  the  process  shown  in  Figure  10,  the  need  for  and  role  played  by  each 
stage  becomes  clear.  Following  Ford’s  process  reinforces  sound  systems  engineering 
principles,  such  as  maintaining  traceability  to  validated  documents  (e.g.  requirements). 
The  process  diagrammed  in  Figure  10  is  executed  by  the  experimental  construct  shown  in 
Figure  11.  This  is  the  same  construct  alluded  to  in  Figure  6,  which  showed  the  scoping 
process  followed  during  research. 
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Figure  11.  The  Experimental  Construct 


There  are  three  components  in  the  experimental  construct.  The  first  component 
captures  the  interoperability  measurement  by  executing  Ford’s  technique.  This 
component  was  built  with  Matlab.  The  second  component  calculates  MOE  3  and  is 
based  on  the  AFRL  white  paper,  JP  2-01,  UML  representations,  and  JCA-provided 
interoperability  characters.  This  component,  called  the  system  interoperability 
component,  was  built  with  Arena.  It  is  linked  to  the  interoperability  measurement 
component  by  a  common  set  of  interoperability  characters.  Recall  that  one  of  the 
primary  research  goals  is  to  look  for  a  correlation  between  the  interoperability 
measurement  and  the  MOEs  related  to  interoperability. 

During  the  course  of  research,  it  was  determined  that  only  one  of  the  six 
established  MOEs  was  directly  linked  to  interoperability  (MOE  3).  Although  the  other 
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MOEs  are  important  to  AFRL/RY,  they  are  not  part  of  the  correlation  study.  This  is  why 
the  system  interoperability  component  only  captures  one  MOE. 

The  other  MOEs  are  captured  by  the  third  experimental  component  which  is 
called  the  mission  resource  component.  Like  the  system  interoperability  component,  this 
part  of  the  experiment  is  based  on  the  AFRL  white  paper,  JP  2-01,  and  UML 
representations  of  the  layered  sensing  system.  It  is  executed  with  Arena. 

The  two  Arena  components  are  linked  by  a  common  user  interface  (UI),  shown  in 
Figure  12.  This  UI  is  an  Excel  spreadsheet  in  which  the  user  inputs  layered  sensing 
system  features  that  describe  platforms,  sensors,  and  other  elements  of  the  simulation. 
The  Arena  software  reads  these  inputs  as  variables  and  uses  them  to  initialize  the  discrete 
event  simulation. 

Although  linked  by  a  common  user  input,  the  two  Arena  simulation  components 
do  not  use  the  same  process  flow.  Figures  13  and  14  display  the  processes  executed  by 
the  simulation.  Figure  13  shows  the  process  that  captures  MOE  3,  which  is  measured 
with  units  of  time.  This  process  encapsulates  the  ISR  mission  as  described  in  JP  2-01. 
Figure  14  shows  the  process  that  captures  MOEs  1,  2,  4,  5,  and  6.  These  MOEs  are 
related  to  reliability,  coverage,  mission  failures,  and  other  performance  metrics.  These 
MOEs  are  important  for  comparing  different  layered  sensing  architectures,  but  are  not 
directly  related  to  interoperability. 
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Figure  12.  Discrete  Event  Simulation  User  Interface 
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Figure  13.  Simulation  Process  Flow  for  the  System  Interoperability  Component 
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Figure  14.  Simulation  Process  Flow  for  the  Mission  Resource  Component 
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3.4  Measuring  Interoperability 

Ford’s  process  starts  by  stating  a  purpose.  For  the  current  research,  the  general 
purpose  (or  perhaps  interest)  has  at  least  two  elements  linked  to  vetted  DoD 
interoperability  discussions:  measure  system-to-system  compatibility,  and  determine  ISR 
product  distribution  bottlenecks  (4:111-7).  As  the  focus  narrows  to  layered  sensing,  the 
specific  purpose  becomes  generating  quantitative  comparisons  of  different  layered 
sensing  architectures,  with  an  emphasis  on  interoperability. 

The  next  step  is  identifying  a  specific  process.  Starting  once  again  with  a  general 
perspective,  the  natural  candidate  is  outlined  in  JP  2-01,  “Joint  and  National  Intelligence 
Support  for  Military  Operations.”  Moving  towards  a  specific  process  gives  that  outlined 
in  UC  2.  The  two  processes  complement  each  other  and  when  mated  together  provide  the 
specific  ISR  process  mimicked  by  the  simulation. 

MOE  selection  is  next.  The  MOEs  that  are  selected  to  evaluate  system 
performance  must  be  linked  to  the  attributes  drawn  from  the  AFRL  white  paper  that 
characterized  layered  sensing.  The  use  case  chosen  for  emulation  should  also  influence 
MOE  selection.  Recall  that  the  MOEs  are  listed  in  Table  3.  MOE  identification  is  one  of 
three  critical  links  between  the  experimental  components.  The  other  two  links  are 
discussed  next. 

System  identification  is  driven  by  an  understanding  of  the  layered  sensing 
architecture.  For  this  research,  there  are  three  broad  categories  of  systems:  platforms  and 
sensors,  the  CAOC,  and  the  ground  forces  conducting  the  mission  of  interest.  The 
ground  force  characteristics  (e.g.  mission  time)  are  set  to  ensure  the  consistency  of 
comparisons  made  between  different  architectures.  Features  for  the  platforms,  sensors, 
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and  the  CAOC  can  be  changed  from  one  experimental  trial  to  the  next.  Feature  selection 
is  driven  by  the  UI  shown  in  Figure  12. 

Interoperability  character  selection  is  the  next  step.  Successful  selection  requires 
a  good  understanding  of  the  current  layered  sensing  architecture,  knowledge  of  potential 
future  changes  to  the  system,  and  access  to  DoD-validated  descriptions  of 
interoperability.  Joint  Capability  Areas  (JCAs)  and  Joint  Integrating  Concepts  (JICs) 
provided  the  requisite  insight  into  ISR  capabilities  required  by  US  forces.  The 
Battlespace  Awareness,  Command  and  Control,  and  Net-centric  JCAs  were  picked  as  the 
final  sources  for  the  interoperability  characters  used  during  experimentation  (specific 
character  definitions  are  in  Appendix  A2).  Recall  that  Table  4  shows  the  characters  that 
were  selected  and  the  specific  source  from  which  they  came.  Character  changes 
represent  potential  new  capabilities  and  are  sometimes  synonymous  with  changes  in 
architecture.  The  interoperability  characters  used  for  experimentation  are  directional  and 
the  states  are  all  absence/presence  states.  (6:42-45)  Understanding  this  notion  might  be 
enhanced  by  applying  one  of  the  interoperability  views  (IV)  introduced  by  Ford  (6:65-70) 
as  possible  additions  to  DOD  Architectural  Framework  (DoDAF).  In  particular,  the 
interoperability  graph  (IV-2)  seems  appropriate  for  deriving  a  better  understanding  of  the 
involved  systems  and  their  interoperations  as  an  intermediate  step  towards  the  following 
instantiation  of  systems.  Figure  15  is  a  schematic  instantiation  of  an  IV-2  for  the  layered 
sensing  architecture  at  hand. 
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Figure  15.  Interoperability  Graph  (IV-2) 


The  next  step  is  instantiating  the  systems  used  to  produce  the  interoperability 
measurement,  and  essentially  combines  the  systems  and  characters  with  their  state  values. 
Table  6  shows  an  example  of  the  instantiation  process.  In  this  case,  the  baseline 
architecture  for  layered  sensing  is  displayed.  The  interoperability  characters  run 
vertically  on  the  left  in  the  table,  and  the  systems  run  horizontally  along  the  top.  The 
table  is  split  in  the  middle,  “in  order  to  accommodate  the  directionality  of  interoperability 
characters”  (6:43).  The  left  half  shows  the  “transmit/can  actively  do”  states  while  the 
right  half  shows  the  “receive/can  understand  or  work  with”  states.  The  bolded 
interoperability  characters  are  the  elements  under  test  during  the  experiments.  For  the 
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purpose  of  this  work  all  states  were  defined  as  either  being  “absent”  (value  0)  or 
“present”  (value  1),  thereby  allowing  the  use  of  a  specific  interoperability  measurement 
method  on  system  similarity  (Equation  1). 

The  measure  of  interoperability  (7)  can  be  calculated  once  the  systems  have  been 
instantiated  (6:50).  The  appropriate  calculation  for  SimBm  (binary  system  similarity)  is 
given  as 

I  =  SimBjn  =  (i)Zfei(<r'(l)  Ab"(1)).  (1) 

The  resulting  matrix  values  resemble  a  normalized  value  (over  n  characters)  for 
the  similarity  between  two  systems  on  each  character  (expressed  by  the  Boolean  AND 
operator  A)  out  of  the  system  instantiating  sequences  a’  and  a”.  Table  7  shows  the 
results  of  the  measurement  for  the  system  instantiation  given  in  Table  6.  The  transmit 
character  direction  for  the  various  systems  is  shown  on  the  vertical  axis  with  the  receive 
character  direction  shown  on  the  horizontal  axis.  The  individual  interoperability  measures 
are  shown  as  fractions  to  allow  for  easy  reading  of  the  number  of  similarities  out  of  the 
overall  number  of  characters.  By  definition,  a  system  is  not  to  be  considered 
interoperable  with  itself  (6:54),  therefore  the  main  diagonal  values  are  set  to  zero.  For 
this  project  it  can  be  assumed  that  there  is  no  interaction  between  the  individual  sensor 
systems.  Although  the  calculation  provides  values  for  sensor  to  sensor  interoperability 
they  are  to  be  considered  meaningless  for  the  remainder  of  this  project. 

The  interoperability  measurement  calculations  were  generated  by  using  a  Matlab 
routine  that  implemented  Equation  1.  The  code  that  executes  the  routine  is  found  in 
Appendix  A3. 
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Table  6.  System  Instantiation  for  all  Interoperability  Characters 
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At  this  point  in  the  experiment,  the  interoperability  measurements  have  been 
made.  The  specific  example  in  this  section  shows  the  baseline  measurement  (how 
layered  sensing  exists  today),  and  serves  as  the  basis  for  comparison  with  all  of  the 
experimental  trials  that  change  various  characters.  Completing  Ford’s  process  requires 
analysis  and  actions  (in  this  case  actions  are  poised  as  recommendations).  These  two 
steps  are  dealt  with  in  Chapters  4  and  5  and  will  not  be  discussed  here. 

Table  7.  Layered  Sensing  Baseline  Architecture  Interoperability  Measurement 


BASELINE 

LAIR 

ARGUS-IS 

GOTCHA 

NITESTARE 

GENERIC 

CAOC 

GF 

LAIR 

0 

7/12 

1/2 

13/24 

7/12 

5/8 

19/24 

ARGUS-IS 

5/12 

0 

1/3 

3/8 

5/12 

13/24 

13/24 

GOTCHA 

1/2 

1/2 

0 

1/2 

13/24 

13/24 

17/24 

NITESTARE 

13/24 

13/24 

1/2 

0 

7/12 

5/8 

19/24 

GENERIC 

5/12 

5/12 

3/8 

5/12 

0 

5/8 

7/12 

CAOC 

5/12 

5/12 

5/12 

5/12 

11/24 

0 

5/8 

GF 

5/12 

5/12 

5/12 

5/12 

5/12 

5/12 

0 

3.5  Calculating  Measures  of  Effectiveness 

The  process  that  generates  interoperability  measurements  also  guides  construction 
of  the  discrete  event  simulation.  The  starting  point  is  certainly  the  same:  vetted  DoD 
joint  doctrine  and  policy.  The  same  guidance  that  shaped  general  purpose  and  process 
selection  for  Ford’s  technique  also  applies  to  the  simulation:  JCAs,  JICs,  and  JP  2-01. 

For  the  specific  focus,  attention  centers  on  simulating  UC  2  with  the  current  layered 
sensing  architecture.  The  MOEs  calculated  by  this  specific  UC  2  simulation  constitute 
the  baseline  to  which  experimental  trials  will  be  compared.  Each  numerical  experiment 
represents  a  change  in  the  state  of  one  or  more  interoperability  characters  and  calculates 
new  MOE  values. 
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Selecting  MOEs  for  evaluating  performance  is  critical  to  the  experimental 
process.  As  the  simulation  was  built  and  tested,  it  became  obvious  that  only  one  of  the 
MOEs  identified  during  the  execution  of  Ford’s  technique  for  measuring  interoperability 
graded  performance  in  a  way  clearly  linked  to  interoperability  (Table  3,  MOE  3).  This 
observation  led  to  breaking  the  simulation  into  two  parts:  a  part  that  calculates  values  for 
MOEs  not  related  to  interoperability  (but  still  desired  by  the  sponsor)  called  the  mission 
resource  component  and  a  part  that  calculates  the  value  of  the  one  MOE  that  is  related  to 
interoperability  called  the  system  interoperability  component. 

Just  as  with  the  execution  of  Ford’s  technique,  systems  and  interoperability 
characters  must  be  indentified  before  and  during  construction  of  the  simulation.  The 
systems  listed  in  Table  1  are  in  the  discrete  event  simulation,  and  are  characterized  by  the 
UI  spreadsheet  shown  in  Figure  12.  The  bolded  interoperability  characters  shown  in 
Table  6  are  included  in  the  simulation. 

There  was  no  clear  point  where  preparation  stopped  and  simulation  construction 
began:  for  a  significant  period  of  time  both  steps  were  in  progress.  Building  the  model 
became  more  than  just  crafting  an  experimental  tool.  As  much  insight  into 
interoperability  was  gained  while  the  model  was  built  as  was  secured  during  numerical 
experimentation.  Interested  readers  will  find  a  detailed  treatment  of  the  Arena 
simulation  code  in  Appendix  A5. 

The  simulation  must  capture  values  for  MOEs,  especially  the  MOE  related  to 
interoperability.  However,  MOE  values  could  not  be  correlated  to  Ford’s  interoperability 
measurements  unless  interoperability  characters  are  also  addressed.  The  interoperability 
characters  are  represented  by  decision  nodes  in  the  simulation  component  that  calculates 
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the  MOE  related  to  interoperability.  A  given  character  presence  state  (a  “1”  in  Table  6) 
corresponds  to  a  “yes”  in  an  Arena  simulation  decision  node.  A  decision  node  “yes” 
opens  a  new  path  that  a  packet  of  data  (in  the  simulation)  can  follow  as  it  travels  from  a 
sensor  on  a  platform  to  the  ground  forces.  A  “no”  in  a  decision  node  means  the  data 
packet  must  reach  the  ground  forces  by  a  different  path.  So  decision  nodes  are  not  only 
important  because  they  provide  the  mechanism  that  allows  for  character  linkage  between 
Ford’s  technique  and  the  simulation,  they  also  drive  one  of  two  simulation  outputs: 
number  of  paths  that  data  can  follow  through  the  ISR  process  in  order  to  get  from  sensor 
to  ground  forces.  For  the  duration  of  this  report,  these  paths  are  called  “process  paths.” 

The  simulation  is  demonstrated  with  the  baseline  performance  output.  Recall  that 
baseline  means  layered  sensing  system  characterization  as  the  architecture  is  currently 
defined.  Figure  16  shows  the  time  it  takes  for  data  to  travel  from  the  sensor/platform  to 
the  ground  forces.  Recall  that  there  are  five  sets  of  histograms  shown  in  Figure  16 
because  the  simulation  contains  five  different  sensor/platform  systems.  Three  different 
results  are  graphed  for  each  named  system.  The  simulation  ran  250  times  to  characterize 
the  baseline  (all  experimental  trials  were  executed  either  250  or  500  times),  so  the 
average  time  it  takes  for  data  gathered  by  a  system  to  travel  to  the  ground  forces  is 
graphed.  This  graph  also  has  bars  denoting  the  95%  confidence  interval.  The  other  two 
graphed  outputs  are  maximum  average  and  minimum  average  times  a  data  packet  from  a 
given  system  needed  to  traverse  the  available  process  paths  to  the  ground  forces. 

The  results  from  six  experimental  runs  will  be  presented  in  Chapter  4.  Although 
the  display  format  remains  constant,  these  time  plots  are  different  than  that  shown  for  the 
baseline  performance  because  they  show  a  time  difference  (vice  a  time  of  completion). 
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The  difference  is  (baseline  -  experiment).  So  a  plot  with  a  positive  time  value  implies 
better  performance  for  the  experimental  interoperability  character  state  change.  A 
negative  time  value  implies  that  the  baseline  performed  better. 


Figure  16.  Baseline  Performance  —  Time  for  Data  to  Travel  from  Sensor/Platform 

to  Ground  Forces 

Figure  17  records  the  number  of  process  paths  that  data  from  each  system  can 
take  to  reach  the  ground  forces  for  the  baseline  layered  sensing  architecture.  The 
corresponding  plots  for  the  experimental  trials  shown  in  Chapter  4  graph  two  numbers  for 
each  system:  number  of  process  paths  before  and  after  the  interoperability  character  state 
was  changed  (baseline  path  number  and  experimental  path  number).  Figure  16  and 
Figure  17  are  the  outputs  from  the  simulation  component  that  captures  the  MOE  related 
to  interoperability.  Figure  16,  time  for  data  to  travel  from  sensor  to  ground  forces,  is  a 
plot  of  the  MOE.  The  mission  resource  simulation  component  produces  plots  with  a 
different  format  than  that  shown  in  Figure  16  and  Figure  17,  but  the  presentations  are 
straight-forward  and  do  not  require  any  preparatory  discussion  outside  of  Chapter  4. 
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Figure  17.  Baseline  Performance  --  Number  of  Process  Paths  Data  Can  Take 
between  Sensor  and  Ground  Forces 


3.6  Linking  Interoperability  Measurements  and  Measures  of  Effectiveness 

One  of  the  research  goals  for  this  project  was  testing  for  correlation  between 
Ford’s  interoperability  measurements  and  discrete  event  simulation  captured-MOE 
values.  Pursuit  of  this  goal  requires  a  balancing  act  between  the  execution  of  Ford’s 
technique  and  the  simulation  trials.  The  two  experimental  components  must  have  parallel 
constructs  so  that  meaningful  comparisons  can  be  made.  However,  input  used  for  one 
component  cannot  improperly  influence  the  other  component’s  performance. 

There  are  three  elements  in  place  to  prevent  improper  influence:  interoperability 
characters,  character  state  values,  and  MOEs.  The  selection  of  interoperability  characters 
serves  as  the  focal  point  for  balance.  The  initial  list  of  potential  characters  came  from 
DoD  doctrine  and  policy.  As  UC  2  began  to  unfold  in  the  Arena  simulation  the 
researchers  gained  insight  into  what  characters  were  relevant.  Reality  -  or  at  least  reality 
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as  it  is  represented  by  the  simulation  -  drove  character  selection.  The  final  list  of 
characters  that  matured  from  the  simulation  was  imported  for  use  in  Ford’s 
interoperability  measurement  technique. 

Once  the  characters  were  set,  an  initial  character  state  configuration  was  required. 
The  initial  state  values  (Os  or  Is)  were  driven  by  the  current  layered  sensing  architecture. 
An  experiment  constitutes  a  potential  future  capability  change  for  the  layered  sensing 
system  of  systems.  This  potential  change  in  capability  is  represented  by  a  change  in  state 
value  for  at  least  one  interoperability  character.  The  change  of  state  links  a  specific 
interoperability  measurement  to  a  specific  MOE  value. 

The  MOEs  are  linked  to  and  driven  by  the  layered  sensing  attributes  described  by 
AFRL;  the  MOEs  are  the  performance  yardstick  for  all  components  of  the  simulation. 

An  increase  in  interoperability  measurement  is  not  considered  an  architecture 
improvement  unless  there  is  a  corresponding  improvement  in  MOE  value.  The  AFIT 
researchers  think  the  way  that  characters,  state  values,  and  MOEs  were  determined,  used, 
and  calculated  both  ensure  the  proper  linkage  between  the  experimental  components  and 
limit  any  undue  influence  between  interoperability  measurement  and  MOE  calculation. 

There  is  one  final  note  regarding  MOEs.  Calculating  a  MOE  for  specific  systems 
stretches  the  definition  of  MOE  in  an  undesirable  direction:  towards  that  of  a  MOP.  The 
researchers  realize  this,  but  the  need  for  MOEs  and  MOPs  came  about  during 
experimentation  and  will  be  discussed  in  detail  in  Chapter  4. 
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3.7  Experimental  Test  Plans 

The  number  of  potential  interoperability  character  state  changes  allows  for  a  large 
number  of  experiments,  even  when  interest  is  constrained  to  interoperability.  This  rich 
experimental  environment  was  the  design  intent:  more  experiments  provide  more  insight 
into  different  layered  sensing  architectures.  Including  a  complete  listing  of  potential 
experiments  would  be  cumbersome,  so  two  test  plans  are  provided  for  the  experiments 
discussed  in  Chapter  4  and  they  stand  as  surrogate  examples  for  other  possible  executions 
of  the  methodology.  The  two  plans  test  both  parts  of  the  simulation. 

Table  8  shows  the  test  plan  for  exploring  correlation  between  interoperability 
measurements  and  MOE  calculations.  It  captures  a  complete  set  of  experiments  needed 
to  explore  the  core  research  goal. 


Table  8.  System  Interoperability  Test  Plan 


Trial  Number 

Interoperability 
Character  under  Test 

Rationale 

IE  1 

2. 1.2. 3. 1.2  Infrared 
Collection 

Add  IR  sensor  to  Argus 
system  . . .  quantify 
interoperability  impact,  take 
advantage  of  Argus  loiter 
time,  provide  day /night 
coverage 

IE  2 

6. 1.2. 2  Beyond  Line  of 
Sight  (BLOS) 

Add  BLOS  capability  to 
ground  forces  . . .  quantify 
interoperability  impact 
when  CAOC  talks  directly 
to  ground  force  element 

IE  3 

6. 1.2. 2  BLOS 

Add  BLOS  capability  to  the 
Argus  system  . . .  quantify 
interoperability  impact 
when  CAOC  and  platform 
no  longer  have  to  be  within 
line-of-sight  (LOS) 
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IE  4 

6. 1.2.1  LOS 

Move  CAOC  within  LOS  of 
the  AOR  (<  175  km)  . . . 
does  not  directly  change 
any  character,  but  impacts 
LOS  communication  for 
layered  sensing  system 

IE  5 

2. 1.2. 3. 1.2  Infrared 
Collection 

6. 1.2. 2  BLOS 

Add  two  interoperability 
characters  for  the  Argus 
system  . . .  quantify 
interoperability  from  mixed 
characters  state  impacts 

IE  6 

2.1.4  Analysis  and 
Production 

2.1.5  Intelligence, 
Surveillance  and 
Reconnaissance 
Dissemination 

Add  two  interoperability 
characters  for  the  generic 
system. . .  quantify  impact 
of  significant 
interoperability 
improvement 

Table  9  shows  a  sampling  of  simulation  experiments  that  characterize  different 
layered  sensing  architectures.  Although  not  linked  to  interoperability  measurements, 
these  MOE  comparisons  can  support  future  acquisition  decisions  affecting  layered 
sensing. 


Table  9.  Mission  Resource  Experimental  Test  Plan 


Trial  Number 

Measure  of  Effectiveness 
under  Test 

Action 

ME  1 

Potentially  MOEs  1,  2,  4,  5, 
and  6 

Add  IR  sensor  to  Argus 

ME  2 

Potentially  MOEs  1,  2,  4,  5, 
and  6 

Lower  aircraft  reliability  for 
generic  platform 

ME  3 

Potentially  MOEs  1,  2,  4,  5, 
and  6 

Change  from  summer  to 
winter,  decrease  in  amount 
of  daylight 

ME  4 

Potentially  MOEs  1,  2,  4,  5, 
and  6 

Decrease  number  of 
platforms  available  for 
tasking 
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4.  Analysis  and  Results 


4.1  Overview 

This  chapter  contains  the  results  from  all  components  of  the  experimental 
construct.  The  possible  correlation  between  interoperability  measurement  and  MOE  3  is 
explored  first.  The  next  section  describes  the  concept  of  an  ISR  process  path  and  its 
potential  impact  on  interoperability.  Next,  the  difference  between  MOE  and  MOP  as  it 
relates  to  this  research  effort  is  addressed.  The  chapter  concludes  with  the  mission 
resource  MOE  experiments. 

4.2  Interoperability  Measurements  and  Measures  of  Effectiveness 

This  section  is  organized  by  system  interoperability  experiment  (reference 
Table  8).  First,  the  interoperability  measurement  for  the  baseline  configuration  of  layered 
sensing  is  shown  again  in  Table  10  so  that  the  reader  does  not  need  to  search  for  the 
earlier  reference. 


Table  10.  Baseline  Interoperability  Measurement 


LAIR 

ARGUS-IS 

GOTCHA 

NITESTARE 

GENERIC 

CAOC 

GF 

LAIR 

0 

7/12 

1/2 

13/24 

7/12 

5/8 

19/24 

ARGUS-IS 

5/12 

0 

1/3 

3/8 

5/12 

13/24 

13/24 

GOTCHA 

1/2 

1/2 

0 

1/2 

13/24 

13/24 

17/24 

NITESTARE 

13/24 

13/24 

1/2 

0 

7/12 

5/8 

19/24 

GENERIC 

5/12 

5/12 

3/8 

5/12 

0 

5/8 

7/12 

CAOC 

5/12 

5/12 

5/12 

5/12 

11/24 

0 

5/8 

GF 

5/12 

5/12 

5/12 

5/12 

5/12 

5/12 

0 
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The  discrete  event  simulation  produced  six  sets  of  experimental  results  for  system 
interoperability  testing.  Four  sets  are  discussed  in  this  section.  Experiments  IE  3  and 
IE  5  (see  Table  8)  are  not  covered  because  they  do  not  add  insight  beyond  that  provided 
by  the  four  experiments  that  are  presented.  For  a  given  experiment,  one  table  and  one  or 
two  figures  may  be  shown  for  discussion.  The  table  shows  both  the  interoperability 
measurement  and  changes  from  the  baseline  measurement  (in  parentheses).  The  figures 
either  show  average  (with  95%  confidence  intervals),  minimum  average,  and  maximum 
average  MOE  values  or  the  number  of  process  paths.  Analytical  comments  are  provided 
for  each  experiment.  Figures  are  only  included  if  they  are  necessary  for  the  discussion. 

The  primary  goal  for  these  experiments  was  to  look  for  a  statistical  correlation 
between  interoperability  measurement  and  MOE  3.  The  results  show  that  as  far  as  the 
simulated  scenario  is  concerned,  no  correlation  exists.  Despite  this  finding,  several  other 
useful  observations  were  made  and  will  be  discussed  in  this  chapter  and  Chapter  5. 
Experiment  IE  1:  Argus  System  Receives  IR  Capability 
Analytical  Comments: 

In  this  experiment,  an  interoperability  character  state  was  changed  so  that  the 
Argus  system  would  have  a  new  IR  sensor.  Table  11  shows  that  the  interoperability 
measurement  did  increase,  albeit  slightly.  This  result  means  that  according  to  Ford’s 
technique  the  layered  sensing  architecture  in  IE  1  is  more  interoperable  than  the 
architecture  in  the  baseline.  One  can  consider  Argus  more  common  (shared  capability) 
with  Nitestare  and  the  Generic  system.  However,  the  simulation  results  for  MOE  3 
shown  in  Figure  18  show  no  significant  improvement  in  effectiveness.  This  is  to  be 
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expected,  since  the  interoperability  character  selected  for  experimentation  did  not 
increase  the  number  of  ISR  process  paths  present  in  the  architecture  (see  Figure  19). 

This  experiment  accomplishes  two  tasks.  First,  and  most  importantly,  it  shows 
that  a  change  in  interoperability  measurement  is  not  always  accompanied  by  a  significant 
change  in  mission  effectiveness.  Second,  it  helps  validate  simulation  performance.  The 
values  for  MOE  3  should  not  change  significantly  unless  the  number  of  process  paths 
changes,  so  IE  1  shows  that  the  simulation  is  performing  as  designed. 

Table  11.  Experiment  IE  1  Interoperability  Measurement 


LAIR 

ARGUS-IS 

GOTCHA 

NITESTARE 

GENERIC 

CAOC 

GF 

LAIR 

0 

7/12 

1/2 

13/24 

7/12 

5/8 

19/24 

ARGUS-IS 

5/12 

0 

1/3 

5/12  (1/24) 

11/24  (1/24) 

7/12  (1/24) 

7/12  (1/24) 

GOTCHA 

1/2 

1/2 

0 

1/2 

13/24 

13/24 

17/24 

NITESTARE 

13/24 

7/12  (1/24) 

1/2 

0 

7/12 

5/8 

19/24 

GENERIC 

5/12 

11/24  (1/24) 

3/8 

5/12 

0 

5/8 

7/12 

CAOC 

5/12 

5/12 

5/12 

5/12 

11/24 

0 

5/8 

GF 

5/12 

5/12 

5/12 

5/12 

5/12 

5/12 

0 

Figure  18.  Experiment  IE  1  Measure  of  Effectiveness 
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Figure  19.  Experiment  IE  1  Path  Number  Comparisons 

Experiment  IE  2:  Ground  Forces  Receive  Beyond  Line  of  Sight  Communications 
Analytical  Comments: 

In  this  experiment  an  interoperability  character  state  was  changed  so  that  the 
ground  forces  would  have  a  new  capability  for  communicating  with  the  generic  system 
and  the  CAOC.  The  character  change  caused  an  improvement  in  the  interoperability 
measurement  (see  Table  12).  The  impact  on  generic  system  performance  was  mixed, 
however  (see  Figure  20). 

Table  12.  Experiment  IE  2  Interoperability  Measurement 


LAIR 

ARGUS-IS 

GOTCHA 

NITESTARE 

GENERIC 

CAOC 

GF 

LAIR 

0 

7/12 

1/2 

13/24 

7/12 

5/8 

19/24 

ARGUS-IS 

5/12 

0 

1/3 

3/8 

5/12 

13/24 

13/24 

GOTCHA 

1/2 

1/2 

0 

1/2 

13/24 

13/24 

17/24 

NITESTARE 

13/24 

13/24 

1/2 

0 

7/12 

5/8 

19/24 

GENERIC 

5/12 

5/12 

3/8 

5/12 

0 

5/8 

5/8  (1/24) 

CAOC 

5/12 

5/12 

5/12 

5/12 

11/24 

0 

2/3  (1/24) 

GF 

5/12 

5/12 

5/12 

5/12 

11/24  (1/24) 

11/24  (1/24) 

0 
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The  explanation  for  mixed  performance  rests  with  the  change  in  process  paths 
(see  Figure  21).  Recall  that  the  generic  system  has  beyond-line-of-sight  communications 
as  a  default  configuration.  As  such,  giving  this  same  capability  to  the  ground  forces 
increased  the  number  of  process  paths  for  the  generic  system  from  2  to  18.  The 
significant  increase  in  number  of  process  paths  makes  it  difficult  to  compare  MOE  3 
results  between  the  layered  sensing  baseline  configuration  and  that  contained  in  IE  2. 
Unless  MOE  3  values  are  tracked  for  each  process  path,  it  is  difficult  to  identify  specific 
performance  concerns. 

These  results  further  demonstrate  that  an  improvement  in  interoperability 
measurement  may  not  clearly  or  directly  translate  into  an  increase  in  mission 


effectiveness  -  at  least  not  when  evaluating  architectures  at  this  level  of  detail. 


Figure  20.  Experiment  IE  2  Measure  of  Effectiveness 

This  experiment  also  helps  validate  simulation  performance.  The  researchers 
would  only  expect  significant  changes  in  MOE  3  values  for  systems  that  saw  a  change  in 
the  number  of  process  paths.  In  this  case,  significant  MOE  3  changes  were  observed  for 
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the  generic  system  but  not  for  the  other  four.  This  observation  is  consistent  with  the 
change  in  process  path,  so  the  discrete  event  simulation  is  performing  as  designed. 
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Figure  21.  Experiment  IE  2  Path  Number  Comparisons 

Experiment  IE  4:  The  CAOC  System  Is  Placed  within  Line  of  Sight  of  the  AOR 
Analytical  Comments: 

No  interoperability  character  states  were  changed  in  this  experiment,  so  the 
interoperability  measurement  stayed  constant.  Moving  the  CAOC  to  within  line-of-sight 
of  the  AOR  represents  an  operational  change.  The  intent  behind  IE  4  was  to  see  if 
MOE  3  would  change  significantly  when  the  interoperability  measurement  stayed 
constant.  Figure  22  shows  that  at  least  some  aspects  of  mission  effectiveness  (average 
time,  minimum  average  time,  or  maximum  average  time)  varied  significantly.  This 
observation  is  confirmed  by  Figure  23,  which  shows  a  marked  increase  in  the  number  of 
process  paths  for  all  systems.  This  experiment  shows  that  MOE  3  can  change  regardless 
of  what  happens  to  the  interoperability  measurement,  further  weakening  the  case  for  a 
consistent  link  between  interoperability  and  mission  effectiveness. 
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Figure  22.  Experiment  IE  4  Measure  of  Effectiveness 


Figure  23.  Experiment  IE  4  Path  Number  Comparisons 
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Experiment  IE  6:  The  Generic  System  Receives  a  Man-in-the-Loop 
Analytical  Comments: 

For  this  experiment  two  interoperability  character  states  were  changed  so  that  the 
generic  system  would  have  new  analysis  and  dissemination  capabilities  that  simulated  a 
man-in-the-loop  on  the  platform.  Table  13  shows  that  the  interoperability  measurement 
improved  as  the  generic  system  is  more  common  with  the  CAOC  and  the  ground  forces. 
Figure  24  shows  that  mission  effectiveness  for  the  generic  system  also  improved  for  all 
three  plotted  time  values.  Figure  25  shows  that  the  number  of  process  paths  increased 
from  2  to  4. 


Table  13.  Experiment  IE  6  Interoperability  Measurement 


LAIR 

ARGUS-IS 

GOTCHA 

NITESTARE 

GENERIC 

CAOC 

GF 

LAIR 

0 

7/12 

1/2 

13/24 

7/12 

5/8 

19/24 

ARGUS-IS 

5/12 

0 

1/3 

3/8 

5/12 

13/24 

13/24 

GOTCHA 

1/2 

1/2 

0 

1/2 

13/24 

13/24 

17/24 

NITESTARE 

13/24 

13/24 

1/2 

0 

7/12 

5/8 

19/24 

GENERIC 

5/12 

5/12 

3/8 

5/12 

0 

17/24  (1/12) 

2/3  (1/12) 

CAOC 

5/12 

5/12 

5/12 

5/12 

11/24 

0 

5/8 

GF 

5/12 

5/12 

5/12 

5/12 

5/12 

5/12 

0 

IE  6  shows  that  even  though  improvements  in  the  interoperability  measurement 
may  coincide  with  an  increase  in  mission  effectiveness,  the  magnitude  of  change  in 
interoperability  may  not  match  that  seen  in  effectiveness.  Interoperability  measurement 
changes  on  the  order  of  one  percent  may  occur  in  experiments  that  show  a  significant 
increase  in  the  number  of  process  paths.  The  change  in  path  number,  in  turn,  has 
ramifications  for  the  MOE  3  values  (explored  in  the  next  section). 
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Figure  24.  Experiment  IE  6  Measure  of  Effectiveness 


The  results  from  these  experiments  support  one  conclusion:  an  improvement  in 
interoperability  measurement  does  not  guarantee  an  improvement  in  mission 
effectiveness.  Further,  the  extent  to  which  the  two  are  related  cannot  be  determined  until 
both  components  of  the  experiment  are  performed.  In  other  words,  a  change  in  one 
category  cannot  be  used  to  predict  the  extent  of  change  in  the  other. 


Figure  25.  Experiment  IE  6  Path  Number  Comparisons 
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This  does  not  mean  that  the  interoperability  measure  is  flawed;  it  only  means  that 
the  relationship  between  interoperability  and  mission  effectiveness  is  more  complex  than 
originally  thought.  Other  MOEs  linked  to  interoperability  may  have  showed  different 
degrees  of  linkage.  Interoperability  character  state  representations  more  complex  than 
the  utilized  Boolean  scheme  may  also  influence  the  results. 

4.3  Process  Paths 

Failure  to  show  a  correlation  between  the  interoperability  measurement  and  the 
MOE  linked  to  interoperability  was  disappointing,  at  least  initially.  However,  the  effort 
put  into  the  investigation  did  lead  to  a  deeper  appreciation  for  the  significance  of  process 
paths.  This  new  appreciation  produced  a  significant  finding  which  is  described  in  this 
section. 

Recall  that  in  experiment  IE  6  the  generic  system  was  given  the  capability  to 
analyze  and  disseminate  data.  This  capability  change  increased  the  number  of  process 
paths  from  2  to  4.  Figure  26  and  Figure  27  show  the  baseline  and  IE  6  process  paths  for 
the  generic  system. 

The  new  generic  system  capabilities  allowed  more  of  the  ISR  process  to  be 
completed  on  the  platform.  As  was  the  design  intent  during  discrete  event  simulation 
construction,  performing  more  steps  of  the  ISR  process  on  the  platform  decreased  the 
amount  of  time  it  takes  to  get  data  to  the  ground  forces.  As  Figure  28  and  Figure  29 
show,  the  change  in  mission  performance  can  be  significant. 
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Figure  26.  Baseline  Process  Paths  for  the  Generic  System 


Figure  27.  IE  6  Process  Paths  for  the  Generic  System 
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The  process  path  in  IE  6  (path  4  in  Figure  28)  that  took  the  shortest  time  to 


complete  was  approximately  6  minutes  40  seconds  faster  than  the  fastest  process  path  in 
the  baseline  (path  2  in  Figure  28).  Interpreting  this  result  literally  means  the 
experimental  architecture  improved  MOE  3  by  42%.  Further,  executing  a  2-sample 
T-distribution  test  with  unequal  variances  and  365  degrees  of  freedom  allows  the 
researchers  to  conclude  that  the  average  times  for  path  2  and  path  4  are  indeed  different 
in  a  statistically  significant  sense,  with  a  confidence  level  exceeding  99.95%.  (10:336  - 
349,  700)  This  experimental  observation  has  interesting  implications  -  not  for  generic 
platform  performance,  but  for  ISR  process  execution. 


Figure  28.  Time  to  Get  Data  from  System  to  Ground  Forces 

Figure  29  shows  performance  for  paths  2  through  4  as  compared  to  path  1.  The 
data  show  that  layered  sensing  performance  improves  significantly  as  more  of  the  ISR 
process  is  performed  on  the  platform  collecting  the  intelligence  information.  Although 
tempting  to  make,  this  conclusion  is  inappropriate.  The  time  distributions  for  completing 
the  various  ISR  processes  that  were  input  into  the  discrete  event  simulation  are  certainly 
plausible  but  they  may  not  adequately  address  all  possible  scenarios.  As  such,  it  is  better 
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to  use  the  data  to  show  that  process  execution,  in  general,  is  an  important  component  of 
interoperability. 

Building  on  this  point  allows  one  to  consider  that  interoperability,  at  least  for  ISR- 
like  processes,  is  more  than  determining  how  many  sensor  types  should  go  on  any  one 
platform  or  what  type  of  communications  capabilities  a  platform  should  have. 
Interoperability  is  more  than  system  design;  mission  success  may  actually  hinge  on 
finding  the  correct  location  in  the  system  of  systems  architecture  for  executing  each  part 
of  the  process. 


Experiment  IE-6  (GENERIC  system  only) 

average  time  (mi 

(path  1  -  path  i 

g 

IDmin.avg. 

Qmax.avg. 

l . '"1 

path  2  path  3  path  4 

Figure  29.  Process  Path  Performance  Differences 

If  one  considers  that  each  step  in  the  ISR  process  captures  or  manipulates  data  in 
a  manner  that  prepares  that  data  for  reception  by  the  next  step,  than  the  efficiency  of  step- 
to-step  coupling  becomes  an  important  component  of  interoperability.  The  results  from 
IE  6  certainly  seem  to  indicate  a  need  for  further  study  and  quantification  of  process  path 
impact  as  part  of  the  overall  effort  to  characterize  interoperability.  It  is  possible  that 
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successful  process  path  design  and  implementation  may  actually  contribute  more  to 
interoperability  than  changes  in  hardware  or  software  configurations. 

4.4  Measures  of  Effectiveness  Versus  Measures  of  Performance 

Selecting  the  correct  MOEs  for  evaluating  layered  sensing  performance  was  an 
important  part  of  the  research  effort.  Despite  the  level  of  effort  put  into  MOE  selection  at 
the  beginning  of  the  project,  the  task  turned  out  to  be  more  challenging  than  initially 
expected.  The  first  challenge  appeared  during  the  early  stages  of  simulation  execution. 

Of  the  6  MOEs  obtained  by  AFIT  and  vetted  by  AFRL/RY,  only  one  (MOE  3)  showed  a 
direct  link  to  interoperability.  This  situation  was  tolerable  for  two  reasons.  First,  the 
other  MOEs  may  not  have  been  related  to  interoperability  but  they  were  still  important  to 
AFRL.  Second,  time  for  data  to  travel  from  collection  by  the  sensor  to  action  by  the 
ground  forces  (MOE  3)  was  a  solid  benchmark  metric  for  layered  sensing  performance. 

As  the  discrete  event  simulation  matured  it  became  clear  that  capturing  MOE  3  in 
a  meaningful  manner  was  going  to  present  another  challenge.  Ideally,  the  researchers 
wanted  a  MOE  that  captured  mission  effectiveness  for  all  of  layered  sensing,  not  just 
platforms  or  sensors  that  were  part  of  the  architecture.  Previous  depictions  of  MOE  3 
have  listed  average,  minimum  average,  and  maximum  average  time  values  for  each  of  the 
five  systems  included  in  the  layered  sensing  architecture.  For  purposes  of  this  discussion, 
Figure  30  sums  the  times  for  all  systems  into  one  composite  plot  of  average,  minimum 
average,  and  maximum  average  times  for  the  baseline  configuration  and  IE  6. 

Inspection  of  this  figure  shows  little  difference  in  mission  effectiveness.  Granted, 
the  average  performance  for  the  experimental  architecture  improved  by  approximately 
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28  seconds.  This  observation  is  reinforced  by  executing  a  2-sample  T-distribution  test 
with  unequal  variances  and  2269  degrees  of  freedom.  The  test  results  allow  the 
researchers  to  conclude  that  the  average  times  for  the  baseline  and  IE  6  are  indeed 
different  in  a  statistically  significant  sense,  with  a  confidence  level  exceeding  97.5%. 
(10:336  -  349,  700)  Despite  having  a  statistically  significant  difference,  the 
improvement  seems  so  slight  that  architectural  changes  captured  in  IE  6  are  hardly  worth 
considering.  If  inspection  of  IE  6  results  stopped  here,  significant  insight  would  be  lost. 
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Figure  30.  Overall  Performance  for  Layered  Sensing 

Inspection  of  Figure  24  shows  that  IE  6  does  indeed  show  improved  performance, 
at  least  for  the  generic  platform  and  sensor  that  saw  changes  in  interoperability  character 
states.  Although  an  important  observation  had  been  garnered,  it  came  with  a  cost.  This 
cost  was  a  drift  in  measurement  from  MOE  to  MOP.  The  condition  is  further  acerbated 
by  tracking  performance  for  specific  process  paths,  as  shown  in  Figure  28.  Admittedly, 
this  is  not  an  ideal  systems  engineering  construct.  However,  this  deflection  from  the 
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desirable  is  tolerated  because  measuring  performance  for  layered  sensing  as  a  holistic 
entity  did  not  capture  the  fine  detail  needed  to  understand  the  simulation  results.  Had  this 
condition  been  anticipated  at  the  beginning  of  the  effort,  different  approaches  for 
measuring  effectiveness  may  have  been  implemented. 

The  researchers  argue  that  the  use  of  MOPs  has  merit.  Limiting  changes  in 
character  states  for  experimental  trials  to  a  single  system  helped  track  changes  and 
quantify  performance  in  meaningful  ways.  Ultimately,  changes  to  a  single  system  stood 
as  surrogates  for  changes  to  the  entire  architecture.  Experiments  were  run  with 
interoperability  character  state  changes  for  all  systems.  The  result  was  a  large  increase  in 
the  number  of  possible  process  paths  (as  high  as  92  in  some  cases).  Some  of  the  new 
paths  performed  better,  but  many  did  not.  Averaging  across  systems  prevented  the 
researchers  from  identifying  specific  paths  that  performed  better  as  a  result  of  changes  to 
one  or  a  small  number  of  interoperability  character  states.  Valuable  insight  would  have 
been  lost  had  performance  not  been  tracked  by  both  MOE  and  MOP. 

4.5  Measures  of  Effectiveness  for  Mission  Resources 

This  section  presents  data  samples  produced  by  the  mission  resource  experiments 
described  in  Table  9.  The  data  presentation  is  broken  down  by  MOE.  The  results  of 
experiments  ME  1  through  ME  4  are  listed  in  the  plot  shown  for  each  MOE,  as  is  the 
baseline  value.  The  data  shows  how  the  simulation  can  evaluate  layered  sensing  system 
performance  for  MOEs  that  are  not  only  related  to  interoperability  measurements. 
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MOE  1  and  MOE  6:  Percentage  of  Time  Mission  Is  Covered  by  at  Least  One  Platform 
and  Percentage  of  Time  Mission  Covered  by  at  Least  Two  Platforms 

The  line  in  the  center  of  each  blue  bar  in  Figure  31  and  Figure  32  is  the  average 
percentage  of  mission  coverage  by  the  indicated  number  of  sensors.  The  bar  endpoints 
are  average  minimum  and  maximum  values.  The  endpoints  of  the  lines  are  extreme 
minimum  and  maximum  values.  The  95%  confidence  levels  have  been  plotted  for  the 
average  values  shown  on  each  figure. 

Analytical  Comments: 

Figure  31  shows  that  experiments  run  for  MOE  1  do  not  stress  the  layered  sensing 
architecture  with  regards  to  mission  coverage  by  at  least  one  system.  Coverage  by  at 
least  two  different  sensors  -  an  objective  input  by  AFRL/RY  -  is  more  challenging.  In 
the  case  of  experiment  ME  4  (shown  on  Figure  32),  where  the  number  of  systems 
available  for  the  modeled  use  case  dropped  from  15  to  11,  performance  may  not  meet 
mission  needs.  Adding  an  IR  capability  to  an  E/O  system  (ME  1  in  Figure  32)  shows  an 
increase  in  system  performance,  which  is  to  be  expected.  These  observations  are 
reinforced  by  executing  a  2-sample  T-distribution  test  with  unequal  variances  and  434  or 
487  degrees  of  freedom.  The  statistical  analysis  shows  that  the  difference  in  average 
times  between  the  baseline  and  ME  1  and  the  baseline  and  ME  4  are  statistically 
significant,  with  confidence  levels  of  95%.  (10:336  -  349,  700) 
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Figure  31.  Mission  Coverage  by  at  Least  One  Sensor 


Figure  32.  Mission  Coverage  by  at  Least  Two  Different  Sensors 

MOE  4:  Layered  sensing  mission  failures  in  a  24-hour  period 

The  line  in  the  center  of  each  blue  bar  in  Figure  33  is  the  average  number  of 
system  failures  for  a  24-hour  mission.  The  bar  itself  starts  at  the  minimum  average 
failure  value  and  goes  to  the  maximum  average  failure  value.  The  95%  confidence  levels 
are  included  for  the  average  values. 
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Analytical  Comments: 

Figure  33  shows  that  the  conditions  for  experiment  ME  2  caused  a  minor  failure 
rate  change  from  the  baseline.  None  of  the  experiments  showed  a  significant  operational 
change.  Although  this  particular  set  of  experiments  did  not  impact  the  MOE,  other  user- 
defined  experiments  might  cause  significant  changes.  Regardless,  it  is  an  important 
MOE  for  evaluating  system  performance. 
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Figure  33.  Sensor/Platform  System  Failures  per  24-hour  Mission 

MOE  5:  Average  time  taken  to  begin  mission  coverage 

The  line  in  the  center  of  each  blue  bar  in  Figure  34  is  the  average  time  it  takes  for 
two  systems  to  begin  mission  coverage  (an  MOE  objective  supplied  by  the  sponsor).  The 
bar  endpoints  are  average  minimum  and  maximum  values.  The  endpoints  of  the  lines  are 
extreme  minimum  and  maximum  values.  The  95%  confidence  levels  are  plotted  for  the 
average  values  even  though  the  intervals  are  too  small  to  be  seen  on  the  display. 
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Analytical  Comments: 

For  this  MOE,  decreasing  the  total  number  of  systems  available  to  layered  sensing 
from  15  to  11  (experiment  ME  4)  increased  the  amount  of  time  required  to  begin  event 
coverage.  In  fact,  the  time  to  begin  coverage  nearly  doubled  -  a  result  not  likely 
acceptable  to  the  ground  forces.  Analysis  shows  that  the  baseline  and  ME  4  average 
times  are  different  in  a  statistically  significant  sense.  Experiments  such  as  this  may  help 
AFRL/RY  calculate  a  minimum  acceptable  number  of  systems  needed  for  layered 
sensing  success,  given  a  specific  basing  configuration  and  AOR  size. 
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Figure  34.  Average  Time  to  Begin  Mission  Coverage 
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5.  Conclusions  and  Recommendations 


5.1  Overview 

This  chapter  contains  a  concise  discussion  of  the  research  contributions  and 
significance  to  the  broader  DoD  community.  Recommendations  for  future  action  and 
research  are  also  entertained. 


5.2  Research  Contributions 

The  differences  between  the  intended  and  actual  flow  of  research  activity  must  be 
discussed  before  the  contributions  can  be  put  in  proper  context.  Figure  35  shows  the 
intended  activity  flow  for  this  effort.  There  were  two  parallel  processes.  The  first 
process  attempted  to  characterize  a  suspected  correlation  between  interoperability 
measurements  and  system  interoperability  MOE  values.  As  noted  previously,  the 
correlation  never  materialized. 

The  second  process  attempted  to  capture  mission  resource  MOEs.  This  part  of  the 
experiment  worked  as  planned. 


Determine 

Interoperability 

Characters 


Two  Parallel  Processes 


Capture 

Mission 

Resource 

MOEs 


Figure  35.  Intended  Research  Activity  Flow 
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Figure  36  shows  the  actual  flow  of  research  activity.  As  the  first  process  was 
executed,  the  concept  of  a  process  path  formed  and  grew  in  importance.  The  relationship 
between  determining  interoperability  characters,  calculating  interoperability 
measurements,  and  capturing  the  number  of  process  paths  was  well  understood.  The 
influence  of  process  path  on  interoperability  MOEs  and  MOPs  was  not  completely 
characterized  and  requires  further  research. 
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Two  Parallel 
Processes 


Figure  36.  Actual  Research  Activity  Flow 


Having  described  the  activity  flow,  the  research  contributions  can  now  be  listed: 

1 .  The  relationship  between  interoperability  character  identification  and 
measurement  for  collaborative  interoperability  scenarios  was  adequately  demonstrated. 

2.  Interoperability  measurements  do  not  always  reflect  the  same  magnitude  of 
change  as  that  seen  for  number  of  process  paths.  Small  changes  in  interoperability 
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measurement  (caused  by  changes  in  only  one  or  two  interoperability  character  states) 
may  generate  large  changes  in  the  number  of  process  paths  available  to  the  system. 
Further,  large  changes  in  the  number  of  process  paths  may  lead  to  large  changes  in  MOE 
or  MOP  values. 

3.  For  net-centric  processes  like  those  found  in  ISR  systems,  process  paths  may 
be  an  important  component  of  quantitative  interoperability  measurement  techniques. 

4.  The  number  and  type  of  process  paths  present  in  a  system  of  systems  may  be 
reflected  in  MOEs  and  MOPs,  but  this  subject  requires  further  exploration. 

5.3  Research  Significance 

This  research  is  significant  because  it  demonstrates  that  the  interoperability 
measurement  technique  derived  by  Ford  in  2008  can  be  extended  to  cases  involving 
collaborative  interoperability  through  the  use  of  discrete  event  simulations.  Care  must  be 
given  to  the  selection  of  interoperability  characters  and  the  MOEs  used  to  grade  system 
performance. 

Regardless  of  the  constraints  on  this  research  effort,  it  constitutes  a  positive 
contribution  to  the  DoD  effort  to  quantify  interoperability.  The  degree  to  which  current 
and  future  fielded  military  systems  reach  goals  linked  to  interoperability  will  remain 
unknown  unless  methods  for  measuring  interoperability  are  developed,  used,  and 
analyzed.  This  graduate  research  project  demonstrates  progress  towards  this  end. 

5.4  Recommendations  for  Action 

Although  an  overall  application  of  the  interoperability  measurement  method  in  the 
sense  of  evaluating  different  architectures  has  been  demonstrated,  the  requirement  for 
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more  action  on  this  topic  exists.  The  current  effort  could  be  supplemented  by  further 
combining  the  abilities  of  AFIT  and  AFRL:  additional  structure  and  depth  of  system 
characterization  emphasizing  interoperability  measurement  should  be  fostered  by  AFIT 
whereas  AFRL  could  contribute  by  providing  expertise  for  large  scale  visual  simulation. 
The  suite  of  simulation  tools  currently  under  development  by  the  AFRL/RY  is  especially 
attractive.  Instantiation  of  the  interoperability  characters  is  the  vital  link  between  both 
efforts.  The  call  for  action  on  this  topic  must  be  heeded  in  order  to  successfully 
characterize  the  depth  of  interoperability  character  impact  on  selected  MOEs. 

5.4  Recommendations  for  Future  Research 

Recommendations  for  future  research  include  the  following  topics  and  areas  of 
interest: 

1.  Build  discrete  event  simulations  and  determine  appropriate  MOEs  for  additional  use 
cases  of  interest  to  AFRL. 

2.  Explore  ways  to  capture  more  complete  sets  of  interoperability  characters.  One 
possible  avenue  is  adding  levels  of  complexity  to  the  characters. 

3.  Use  or  derive  more  complex  representations  for  interoperability  character  states  (do 
not  rely  entirely  on  presence/absence  representation). 

4.  Craft  more  robust  simulations  and  execute  more  robust  MOE  identification  and 
calculation  schemes.  Incorporate  more  than  one  interoperability  related  MOE. 

5.  Fully  characterize  the  role  played  by  the  process  path  in  quantitative  interoperability 
analyses. 
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5.5  Summary 

The  methodology  described  in  Chapter  3  and  the  results  discussed  in  Chapter  4 
show  that  changes  in  architecture  can  possibly  be  linked  to  changes  in  mission 
effectiveness  some  of  the  time.  Also,  the  role  of  the  process  path  and  its  relationship  to 
interoperability  require  further  exploration.  The  experimental  components  were 
faithfully  executed  for  the  mission  thread  and  scenario  of  interest.  Interoperability 
measurements  and  MOE  values  were  obtained.  Experiments  were  analyzed  and  results 
were  presented.  Taking  into  account  the  project  scope  and  available  resources,  the 
graduate  research  project  is  deemed  a  success. 
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Al.  Use  Cases 


This  appendix  contains  the  three  use  cases  developed  during  discussions  with 
AFRL/RY.  Recall  that  UC  2  was  the  use  case  simulated  with  the  Arena  software. 


Use  Case  ID 

UC  1  (Event  Tip-off  Determination) 

Operational 
Thread  (Scope) 

Urban  Surveillance 

Level 

Tactical 

Primary  Actor 

Platform  crew,  CAOC/JAOC/JOC  personnel,  ground  forces 

Secondary 

Actors 

CFACC/JFACC/JFC,  environment,  threat,  jamming  threat, 
communications,  operating  base  personnel 

Stakeholders 
and  Interests 

Platform  Crew:  Must  be  able  to  successfully  and  safely  operation 
sensor-equipped  platforms,  must  be  able  to  communicate  with  and 
respond  to  tasking  from  CAOC/JAOC/JOC  and  ground  forces 
personnel. 

CAOC/JAOC/JOC  Personnel:  Must  task  layered  sensing  platforms  and 
sensors  to  collect  data  describing  objectives  and  targets  of  interest,  must 
be  able  to  analyze  collected  data,  must  be  able  to  disseminate  findings 
to  ground  forces 

Ground  Forces:  Require  operational  guidance  garnered  from  analysis  of 
data  provided  by  layered  sensing  system 

CFACC/JFACC:  Must  have  capability  to  task  and  receive  data  from 
platforms  and  sensors  to  support  forces  in  theater 

JFC:  Needs  adequate  ISR  capabilities  to  support  operational/theater 
campaign 

Brief 

Description 

The  CFACC/JFACC  tasks  the  CAOC/JAOC/JOC  to  survey  theater 
battlespace.  The  CAOC  issues  an  ops  order  to  the  base  of  operations. 

The  base  of  operations  launches  a  platform  equipped  with  the 
appropriate  sensor.  The  platform  transitions  to  the  surveillance  area. 
Once  in  the  surveillance  area  the  platform  locates  the  objective  and  any 
appropriate  targets.  The  sensor  collects  intelligence  data.  The  platform 
transmits  the  data  to  the  system  ground  site.  The  ground  site  relays 
sensor  data  to  the  CAOC.  The  CAOC  analyzes  the  data.  An  event  tip- 
off  is  observed.  The  analysis  results  are  disseminated  to  the  appropriate 
actors. 

Preconditions 

Infrastructure,  systems,  and  personnel  are  already  in-place  and  ready  to 
conduct  operations 

Post¬ 

conditions 

Intelligence  data  is  stored  and  accessible  by  authenticated  personnel. 
Results  of  data  analysis  are  passed  to  appropriate  secondary  actors. 

Flow  of  Events 

N/A 

Alternative 
Flows  and 
Exceptions 

1.  Mission  deviations  due  to  environmental  constraints 

2.  Mission  deviations  due  to  threats  in  the  operating  area 

3.  Mission  deviations  due  to  communications  jamming  in  the  operating 
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area 

The  various  constraints  may  result  in  either  mission  delay  or 
cancellation.  In  either  case,  contingency  planning  in  the 
CAOC/JAOC/JOC  will  deal  with  deviations  from  the  happy  path. 

Non-behavior 

Requirements 

Primary  metric  is  MOE  3  (see  Table  3,  Chapter  2) 

Special 

Requirements 

N/A 

Technology 
and  Data 
Variation  List 

Interoperability  characters  (see  Table  4,  Chapter  2) 

Use  Case  ID 

UC  2  (Unplanned  Event  Reaction) 

Operational 
Thread  (Scope) 

Urban  Surveillance 

Level 

Tactical 

Primary  Actor 

Platform  crew,  CAOC/JAOC/JOC  personnel,  ground  forces 

Secondary 

Actors 

CFACC/JFACC/JFC,  environment,  threat,  jamming  threat, 
communications,  operating  base  personnel 

Stakeholders 
and  Interests 

Platform  Crew:  Must  be  able  to  successfully  and  safely  operation 
sensor-equipped  platforms,  must  be  able  to  communicate  with  and 
respond  to  tasking  from  CAOC/JAOC/JOC  and  ground  forces 
personnel. 

CAOC/JAOC/JOC  Personnel:  Must  task  layered  sensing  platforms  and 
sensors  to  collect  data  describing  objectives  and  targets  of  interest,  must 
be  able  to  analyze  collected  data,  must  be  able  to  disseminate  findings 
to  ground  forces. 

Ground  Forces:  Require  operational  guidance  garnered  from  analysis  of 
data  provided  by  layered  sensing  system 

CFACC/JFACC:  Must  have  capability  to  task  and  receive  data  from 
platforms  and  sensors  to  support  forces  in  theater 

JFC:  Needs  adequate  ISR  capabilities  to  support  operational/theater 
campaign 

Brief 

Description 

The  CFACC/JFACC  tasks  the  CAOC/JAOC/JOC  to  survey  theater 
battlespace.  The  CAOC  issues  an  ops  order  to  the  base  of  operations. 

The  base  of  operations  launches  a  platform  equipped  with  the 
appropriate  sensor.  The  platform  transitions  to  the  surveillance  area. 
Once  in  the  surveillance  area  the  platform  locates  the  objective  and  any 
appropriate  targets.  The  sensor  collects  intelligence  data.  The  platform 
transmits  the  data  to  the  system  ground  site.  The  ground  site  relays 
sensor  data  to  the  CAOC.  The  CAOC  analyzes  the  data.  The  analysis 
results  are  disseminated  to  the  ground  forces  so  that  actions  can  be 
taken  to  mitigate  impact  from  unplanned  events. 

Preconditions 

Infrastructure,  systems,  and  personnel  are  already  in-place  and  ready  to 
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conduct  operations 

Post¬ 

conditions 

Intelligence  data  is  stored  and  accessible  by  authenticated  personnel. 
Results  of  data  analysis  are  passed  to  appropriate  secondary  actors. 

Flow  of  Events 

N/A 

Alternative 
Flows  and 
Exceptions 

1.  Mission  deviations  due  to  environmental  constraints 

2.  Mission  deviations  due  to  threats  in  the  operating  area 

3.  Mission  deviations  due  to  communications  jamming  in  the  operating 
area 

The  various  constraints  may  result  in  either  mission  delay  or 
cancellation.  In  either  case,  contingency  planning  in  the 
CAOC/JAOC/JOC  will  deal  with  deviations  from  the  happy  path. 

Non-behavior 

Requirements 

Primary  metric  is  MOE  3  (see  Table  3,  Chapter  2) 

Special 

Requirements 

N/A 

Technology 
and  Data 
Variation  List 

Interoperability  characters  (see  Table  4,  Chapter  2) 

Use  Case  ID 

UC  3  (Forensic  Analysis) 

Operational 
Thread  (Scope) 

Urban  Surveillance 

Level 

Tactical 

Primary  Actor 

Platform  crew,  CAOC/JAOC/JOC  personnel,  ground  forces 

Secondary 

Actors 

CFACC/JFACC/JFC,  environment,  threat,  jamming  threat, 
communications,  operating  base  personnel 

Stakeholders 
and  Interests 

Platform  Crew:  Must  be  able  to  successfully  and  safely  operation 
sensor-equipped  platforms,  must  be  able  to  communicate  with  and 
respond  to  tasking  from  CAOC/JAOC/JOC  and  ground  forces 
personnel. 

CAOC/JAOC/JOC  Personnel:  Must  task  layered  sensing  platforms  and 
sensors  to  collect  data  describing  objectives  and  targets  of  interest,  must 
be  able  to  analyze  collected  data,  must  be  able  to  disseminate  findings 
to  ground  forces 

Ground  Forces:  Require  operational  guidance  garnered  from  analysis  of 
data  provided  by  layered  sensing  system 

CFACC/JFACC:  Must  have  capability  to  task  and  receive  data  from 
platforms  and  sensors  to  support  forces  in  theater 

JFC:  Needs  adequate  ISR  capabilities  to  support  operational/theater 
campaign 

Brief 

Description 

The  CFACC/JFACC  tasks  the  CAOC/JAOC/JOC  to  survey  theater 
battlespace.  The  CAOC  issues  an  ops  order  to  the  base  of  operations. 

The  base  of  operations  launches  a  platform  equipped  with  the 
appropriate  sensor.  The  platform  transitions  to  the  surveillance  area. 
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Once  in  the  surveillance  area  the  platform  locates  the  objective  and  any 
appropriate  targets.  The  sensor  collects  intelligence  data.  The  platform 
transmits  the  data  to  the  system  ground  site.  The  ground  site  relays 
sensor  data  to  the  CAOC.  The  CAOC  analyzes  the  data.  The  analysis 
results  are  disseminated  to  the  appropriate  actors. 

Preconditions 

Infrastructure,  systems,  and  personnel  are  already  in-place  and  ready  to 
conduct  operations 

Post¬ 

conditions 

Intelligence  data  is  stored  and  accessible  by  authenticated  personnel. 
Results  of  data  analysis  are  passed  to  appropriate  secondary  actors. 

Flow  of  Events 

N/A 

Alternative 
Flows  and 
Exceptions 

1.  Mission  deviations  due  to  environmental  constraints 

2.  Mission  deviations  due  to  threats  in  the  operating  area 

3.  Mission  deviations  due  to  communications  jamming  in  the  operating 
area 

The  various  constraints  may  result  in  either  mission  delay  or 
cancellation.  In  either  case,  contingency  planning  in  the 
CAOC/JAOC/JOC  will  deal  with  deviations  from  the  happy  path. 

Non-behavior 

Requirements 

Primary  metric  is  MOE  3  (see  Table  3,  Chapter  2) 

Special 

Requirements 

N/A 

Technology 
and  Data 
Variation  List 

Interoperability  characters  (see  Table  4,  Chapter  2) 
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A2.  Joint  Capability  Area  Interoperability  Character  Definitions 

The  following  JCA-provided  definitions  provide  more  meaning  for  the 
experimental  interoperability  characters. 

2  Battlespace  Awareness  -  The  ability  to  understand  dispositions  and  intentions  as  well 
as  the  characteristics  and  conditions  of  the  operational  environment  that  bear  on  national 
and  military  decision-making. 

2.1  Intelligence,  Surveillance  and  Reconnaissance  -  The  ability  to  conduct  activities  to 
meet  the  intelligence  needs  of  national  and  military  decision-makers. 

2.1.2  Collection  -  The  ability  to  obtain  required  information  to  satisfy  intelligence  needs. 

2.1.2.3  Imagery  Collection  -  The  ability  to  obtain  information  from  the  visible  and  non- 
visible  spectrum  based  on  the  likeness  or  visual  presentation  of  any  natural  or  man-made 
feature,  object,  or  activity. 

2.1.2.3.1  Electro-Optical  Imagery  Collection  -  The  ability  to  gather  information  from  a 
visual  presentation  derived  from  the  ultraviolet  through  far  infrared  electromagnetic 
spectrum. 

2.1.2.3.1.1  Panchromatic  Collection  -  The  ability  to  obtain  a  visual  presentation  from 
the  visible  spectrum  of  any  natural  or  man-made  feature,  object,  or  activity. 
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2.1.2.3.1.2  Infrared  Collection  -  The  ability  to  obtain  a  likeness  or  visual  presentation 
from  the  Infrared  spectrum  of  any  natural  or  man-made  feature,  object,  or  activity. 

2.1.2.3.2  RADAR  Imagery  Collection  -  The  ability  to  derive  information  from  a  visual 
presentation  produced  by  recording  radar  waves  from  a  given  object  within  the 
radiofrequency  spectrum. 

2.1.3  Processing  /  Exploitation  -  The  ability  to  transform  collected  information  into 
forms  suitable  for  further  analysis  or  action. 

2.1.3.1  Data  Transformation  -  The  ability  to  select,  focus,  simplify,  tag  and  transform 
overtly  or  covertly  collected  data  into  human  or  machine  interpretable  form  for 
collaboration  across  the  ISR  community  for  further  analysis  or  other  action. 

2.1.3.2  Objective  /  Target  Categorization  -  The  ability  to  identify,  classify  and  verify 
objectives/targets  enabling  further  analysis  or  action. 

2.1.4  Analysis  and  Production  -  The  ability  to  integrate,  evaluate,  and  interpret 
information  from  available  sources  and  develop  intelligence  products  that  enable 
situational  awareness. 
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2.1.5  Intelligence,  Surveillance  and  Reconnaissance  Dissemination  -  The  ability  to 
present  information  and  intelligence  products  that  enable  understanding  of  the  operational 
environment  to  military  and  national  decision-makers. 

5  Command  and  Control  -  The  ability  to  exercise  authority  and  direction  by  a  properly 
designated  commander  or  decision  maker  over  assigned  and  attached  forces  and 
resources  in  the  accomplishment  of  the  mission. 

5.5  Direct  -  The  ability  to  employ  resources  to  achieve  an  objective. 

5.5.2  Task  -  The  ability  to  direct  actions  and  resources. 

5.5.2.1  Synchronize  Operations  -  The  ability  to  arrange  actions  through  established 
links  with  mission  partners  to  ensure  coordination  of  operations. 

5.5.2.2  Issue  Plans  -  The  ability  to  provide  relevant  plans. 

5.5.2.3  Issue  Orders  -  The  ability  to  provide  directives. 

6  Net-Centric  -  The  ability  to  provide  a  framework  for  full  human  and  technical 
connectivity  and  interoperability  that  allows  all  DoD  users  and  mission  partners  to  share 
the  information  they  need,  when  they  need  it,  in  a  form  they  can  understand  and  act  on 
with  confidence,  and  protects  information  from  those  who  should  not  have  it. 
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6.1  Information  Transport  -  The  ability  to  transport  information  and  services  via 


assured  end-to-end  connectivity  across  the  NC  environment. 


6.1.2  Wired  Transmission  -  The  ability  to  transfer  data  or  information  with  an 
electrical/optical  conductor. 


6.1.2.1  Line  of  Sight  -  The  ability  to  exchange  data  or  information  via  electromagnetic 
spectrum  within  line  of  sight. 


6.1.2.2  Beyond  Line  of  Sight  -  The  ability  to  exchange  data  or  information  via 
electromagnetic  spectrum  beyond  line  of  sight. 
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A3.  Matlab  Code  for  Interoperability  Measurement  Calculation 

The  Matlab  code  that  implements  Ford’s  interoperability  measurement  is  shown  below: 

%  Sim(BIN)  for  directional  interoperability  measurement 

clc; 

myxiFORD  =  xlsread('characters',  -1); 

%  opens  file  'characters.xls'  (in  same  folder)  to  allow  for  array  selection  and  input 

[characters, systems]  =  size(myxiFORD); 

%  define  size  of  the  system  instantiation  matrix  Xi 
systems  =  systems/2; 

%  define  number  of  systems  (only  half  due  to  transmit  and  receive  half  of  matrix) 
Imatrix  =  zeros(systems,  systems); 

%  initially  set  Xi  matrix  to  zeros 

for  transmit  =  l:systems 

for  receive  =  systems+l:systems*2 
if  (transmit  ~=  receive-systems) 

%  automatically  leaves  system  to  system  comparison  at  value  zero 
b  1  =  bitand(myxiFORD(  1  :characters,transmit),myxiFORD(  1  icharacters, receive)); 
%  bitwise  AND  comparison 

I  m  at  r  i  x  ( t  ran  s  m  i  t ,  rccc  i  vc- s  y  s  tc  m  s )  =  sum(bl)/(characters); 
end 
end 
end 

disp(’Interoperability  Character  Matrix  (Instantiation  of  S):'); 

disp(' - '); 

disp(myxiFORD); 
disp(' '); 

disp('Matrix  Size:  ');  disp(size(myxiFORD)); 
disp(' '); 

disp('Interoperability  Matrix  I:'); 

disp(' - '); 

disp(Imatrix); 
disp(' '); 

disp('Mean  Interoperability  Value:'); 

disp(' - '); 

disp(sum(sum(Imatrix))/(numel(Imatrix)-systems)); 
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A4.  Comments  on  the  Interoperability  Measurement  Technique 


Consistency  of  system  characters  through  different  levels 

With  regards  to  the  measurement  of  directional  interoperability  Ford  chose  to 
instantiate  a  given  set  of  systems  S  as  £  =  {  Xt,  Xr  I  using  a  hierarchical  breakdown  of 
interoperability  characters.  These  characters  were  methodically  derived  from  several 
publications  and  doctrines  and  successively  described  by  binary  states.  Although  not 
defined  or  stated  by  Ford,  it  is  to  the  understanding  of  this  group  that  the  presence  of  a 
given  character  state  (i.e.  Xi(sj)  =  1)  at  a  lower  level  within  the  hierarchy  automatically 
results  in  the  appropriate  character’s  state  at  the  higher  level(s)  to  be  present  as  well. 
Using  Ford’s  application  on  SEAD  (6:81-92)  this  can  be  explained  along  the  following 
sample: 


The  interoperability  character  “C2”  is  explained  by  “command  &  control  /  be 
commanded  &  controlled”  (6:86,  table  16). 

The  interoperability  character  “C2.Comm”  is  explained  by  “communicate(T)  / 
communicate(R)”  (6:86,  table  16). 

The  interoperability  character  “C2. Comm. Red”  is  explained  by  “communicate(T) 
on  red  channels  /  communicate(R)  on  red  channels”  (6:86,  table  16). 

The  ability  of  IADS1  and  IADS  2  to  communicate  (T  &  R)  on  red  channels  is 
obvious  and  therefore  described  with  the  character  states  xC2.comm.Red(siADsi)  =  1 
and  xc2.comm.Red(siADS2)  =  1  (6:88,  table  17). 

Accordingly  the  character  states  at  the  higher  levels  C2  and  C2.Comm  are  set  to  1 
as  well  (if  a  system  can  communicate  utilizing  a  certain  channel  it  can  clearly  be 
followed  that  the  same  system  is  able  to  communicate  at  all  and  therefore  is  able 
to  partake  in  command  &  control  processes). 

Having  said  this,  the  same  example  utilized  by  Ford  shows  inconsistency  at  several 

places: 
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Intel. ISR.Detect.Blue(R)  for  the  AOC  is  set  to  “1”  and  should  have  resulted  in 
Intel. ISR.Detect(R),  Intel.ISR(R)  and  Intel(R)  being  characterized  as  present  as 
well. 

Fires. 10. InflOps.Mildec(T)  and  Fires. 10. InflOps(T)  for  IADS1  and  IADS2  are  set 
to  “1”  and  should  have  resulted  in  Fires.IO(T)  be  characterized  as  present  as  well. 
Fires.IO.InflOps.Mildec(R)  and  Fires.IO.InflOps(R)  for  ISR,AOC  and  PSP  are  set 
to  “1”  and  should  have  resulted  in  Fires.IO(R)  and  Fires(R)  be  characterized  as 
present  as  well. 

Application  of  these  findings  to  the  Ford  examples  results  in  a  different  SEAD 
instantiation  X  (see  Table  A4.1)  and  in  conclusion  in  an  apparent  different 
interoperability  measurement  (sees  Table  A4.2). 


Table  A4.1.  Applied  consistency  changes  to  SEAD  instantiation  £  (after  6:88) 


transmit 

receive 

OQ 

a: 

ISR 

AOC 

PSP 

IADS1 
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CO 
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5 

OQ 

a: 

ISR 
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C2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

C2.Comm 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 
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1 

1 

1 

1 

0 

0 

1 

1 

1 

1 

0 

0 
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0 

1 

1 

0 

0 

0 

0 

0 

1 

1 

0 

0 
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0 

0 

0 

1 

1 

0 

1 

0 

0 

1 

1 
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0 

1 

0 

0 

1 

1 

0 

0 
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1 

1 
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0 

1 

0 

0 

1 

1 

0 

0 
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1 

1 

1 
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0 

1 

0 

0 

1 

1 

0 

0 
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1 

1 

1 
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0 

1 

0 

0 

1 

1 

0 

0 

1 

1 

0 

0 
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0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

1 

1 
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0 

0 

0 

1 

1 

1 

0 

1 
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1 

1 

1 
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0 

0 

0 

1 

1 

1 

0 

0 

0 

1 

1 

1 
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0 

0 
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1 

0 

0 

0 

0 

0 

1 

1 
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0 

1 

0 

0 

0 

0 

0 

0 

1 

1 
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1 
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0 

0 

0 

0 

0 
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1 
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0 

0 

0 

1 

1 

0 

0 

0 

0 

1 

1 
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0 

0 

0 

0 

1 

1 

0 

.1 

1 

1 

1 

1 
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0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

1 
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0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

1 

Fires  IO.EW  EA  Barrage 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

1 

Fires  10  EW  EA  Reactive 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

1 

Fires  10  InflOps 

0 

0 

0 

0 

1 

1 

0 

1 

1 

1 

0 

0 

Fires  10  InflOps  MILDEC 

0 

0 

0 

0 

1 

1 

0 

1 

1 

1 

0 

0 

Movement&Maneuver 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Protection 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Sustainment 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 
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Table  A4.2.  Directional  interoperability  measurements  after  consistency  changes 


HB 

ISR 

AOC 

PSP 

IADS1 

IADS2 

HB 

0 

1/9 

1/9 

1/9 

2/27 

2/27 

ISR 

1/9 

0 

8/27  (1/9) 

8/27 

2/9 

2/9 

AOC 

1/9 

1/9 

0 

4/27 

2/27 

2/27 

PSP 

1/9 

4/27  (1/27) 

4/27  (1/27) 

0 

7/27 

7/27 

IADS1 

2/27 

7/27  (2/27) 

10/27  (5/27) 

11/27  (1/27) 

0 

10/27  (1/27) 

IADS2 

2/27 

7/27  (2/27) 

10/27  (5/27) 

11/27  (1/27) 

10/27  (1/27) 

0 

Analyzing  Table  4.2,  the  overall  picture  remains  the  same.  Still  the  adversary 
IADS  seems  to  be  more  capable  than  the  blue  PSP:  IC(PSP,  IADS)  =  7/27  <  IC(IADS, 
PSP)  =  1 1/27).  The  change  in  value  of  1/27  is  deemed  marginal  and  therefore  of  no  real 
concern.  On  the  other  hand,  the  interoperability  measurements  for  IADS  on  the  red  side 
and  ISR  and  AOC  on  the  blue  side  show  different  results.  The  advantage  of  ISR  over 
IADS  ( I0(ISR,  IADS)  =  6/27  >  I0(IADS,  ISR)  =  5/27)  has  flipped  in  favor  of  the  red 
IADS  ( IC(ISR,  IADS)  =  6/27  <  IC(IADS,  ISR)  =  7/27).  For  the  relationship  between 
AOC  and  IADS  the  superiority  of  IADS  has  even  doubled  (I0(IADS,  AOC)  =  5/27  is  now 
IC(IADS,  AOC)  =  10/27)  giving  it  an  ample  margin  of  8/27  instead  of  only  3/27. 

A  similar  picture  can  be  derived  from  further  analyzing  the  second  example  on  an 
upgraded  SEAD  (see  Table  A4.3  and  Table  A4.4).  Here  the  system’s  architecture  was 
changed  due  to  the  addition  of  special  capabilities  to  the  PSP.  While  the  relationships 
between  ISR  and  AOC  on  the  blue  side  and  IADS  on  the  red  side  prove  a  similar  effect  as 
described  before,  the  effects  (i.e.  their  interoperability)  of  IADS  on  PSP  now  hint  towards 
an  even  less  degree  than  before,  suggesting  that  the  additional  capabilities  might  have 
more  impact  on  IADS  than  anticipated  (IG(  IADS,  PSP)  =  9/27  is  now  just  IC(IADS, 

PSP)  =  7/27,  giving  PSP  a  margin  of  5/27  instead  of  just  3/27  over  IADS). 
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Table  A4.3.  Upgraded  SEAD  instantiation  Xu  after  consistency  changes  (after  6:91) 


transmit 

receive 

© 

a: 

ISR 

AOC 

PSP 

IADS1 

IADS2 

OQ 

a: 

ISR 

oov 

PSP 

IADS1 

IADS2 

C2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1  1 

1 

C2  Comm 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1  1 

1 

C2. Comm. Blue 

1 

1 

1 

1 

0 

0 

1 

1 

1 

1  0 

0 

C2.Comm  Blue  Target 

0 

1 

1 

0 

0 

0 

0 

0 

1 

1  0 

0 

C2.Comm.Red 

0 

0 

0 

0 

1 

1 

0 

1 

0 

0  1 

1 

Intel 

0 

1 

0 

0 

1 

1 

0 

0 

1 

o  !  i 

1 

Intel. ISR 

0 

1 

0 

0 

1 

1 

0 

0 

1 

o  j  i 

1 

Intel. ISR.Detect 

0 

1 

0 

0 

1 

1 

0 

0 

1 

o  !  1 

1 

Intel.lSR  Detect  Blue 

0 

1 

0 

0 

1 

1 

0 

0 

1 

0  0 

0 

Intel. ISR  Detect. Red 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0  1 

1 

Fires 

0 

0 

0 

1 

1 

1 

0 

1 

1 

1  1 

1 

Fires  CA 

0 

0 

0 

1 

1 

1 

0 

0 

0 

1  1 

1 

Fires. CA.OCA 

0 

0 

0 

1 

0 

0 

0 

0 

0 

1  1 

1 

Fires  CA  OCA  Ground 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0  1 

1 

Fires. CA.OCA.Ground. Cluster 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0  1 

1 

Fires.  CA.OCA.Ground.  Precision 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0  1 

1 

Fires  CA.DCA 

0 

0 

0 

0 

1 

1 

0 

0 

0 

0  1 

1 

Fires  10 

0 

0 

0 

1 

i  1 

1 

0 

1 

1 

1J  1 

1 

Fires  10  EW 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0  1 

1 

Fires  IO.EW.EA 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0  1 

1 

Fires  10. EW.EA  Barrage 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0  0 

0 

Fires  10  EW  EA.  Reactive 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0  1 

1 

Fires  lO.InflOps 

0 

0 

0 

0 

1 

1 

0 

1 

1 

1  0 

0 

Fires  10. InflOps  MILDEC 

0 

0 

0 

0 

1 

1 

0 

1 

1 

1  0 

0 

Movement&Maneuver 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

Protection 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

Sustainment 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

Table  A4.4.  Directional  interoperability  measurements  for  the  upgraded  SEAD 
instantiation  after  consistency  changes 


HB 

ISR 

AOC 

PSP 

IADS1 

IADS2 

HB 

0 

1/9 

1/9 

1/9 

2/27 

2/27 

ISR 

1/9 

0 

8/27  (1/9) 

4/27  (-1/9) 

2/9 

2/9 

AOC 

1/9 

1/9 

0 

4/27 

2/27 

2/27 

PSP 

1/9 

5/27  (2/27) 

5/27  (2/27) 

0 

4/9 

4/9 

IADS1 

2/27 

7/27  (2/27) 

10/27  (5/27) 

7/27  (-2/27) 

0 

10/27  (1/27) 

IADS2 

2/27 

7/27  (2/27) 

10/27  (5/27) 

7/27  (-2/27) 

10/27  (1/27) 

0 
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To  be  clear,  it  is  not  the  bare  numbers  that  should  attract  the  impact  of  keeping 
consistency  in  the  instantiation  of  interoperability  characters.  Focus  should  rather  be 
guided  towards  the  relative  change  in  numbers  and  the  thorough  analysis  of  each  system 
pair’s  interoperability  measurement. 

The  resulting  character  set  never  the  less  should  pass  through  all  chosen  levels 
from  bottom  to  the  top  to  assign  weight  to  a  given  capability  at  a  lower  level,  i.e.  higher 
degree  of  capability.  For  example  the  characters  C2  and  C2.Comm  show  no  difference  at 
all  over  all  systems  chosen  (all  are  set  to  “1”).  To  dispose  of  one  of  the  two  characters 
(the  highest  level  character  C2  would  seem  to  be  the  logical  choice)  or  even  both  (since 
the  systems  are  exclusively  defined  in  their  character  states  at  the  lower  3rd  and  4th  level 
already)  would  seem  appropriate  but  would  in  the  same  instance  diminish  the  fact  that  the 
systems  are  interoperating  at  a  higher  degree. 

The  consistency  from  lower  levels  to  higher  levels  in  the  chosen  character 
hierarchy,  as  well  as  keeping  higher  levels  to  ensure  weighing  of  higher  degrees  of 
interoperability,  has  been  applied  in  the  present  work. 

Independence  of  interoperability  measurement  from  system  quantities 

In  order  to  find  broad  understanding  in  the  group’s  effort  of  applying  Ford’s 
methodology  on  interoperability  measurement  it  was  necessary  to  evaluate  possible  side 
effects  of  the  number  of  systems  in  a  given  architecture  on  the  resulting  interoperability 
measurement.  Looking  at  Ford’s  example  on  SEAD  (6:81-92),  it  could  not  clearly  be 
understood  why  two  adversary  IADS  systems  (IADS1  and  IADS2)  were  instantiated  but 
only  one  friendly  (blue)  system  each  (HB,  ISR,  AOC,  PSP).  Neither  the  text  nor  the 
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graphs  provided  were  able  to  answer  the  question.  Therefore  the  group  analyzed  a 
different  architecture  by  simply  deleting  IADS2  from  the  set  of  systems  and  recalculating 
the  interoperability  measurement  (see  Table  A4.5). 


Table  A4.5.  Directional  interoperability  measurements  with  only  one  IADS 


HB 

ISR 

AOC 

PSP 

IADS1 

HB 

0 

1/9 

1/9 

1/9 

2/27 

ISR 

1/9 

0 

5/27 

8/27 

2/9 

AOC 

1/9 

1/9 

0 

4/27 

2/27 

PSP 

1/9 

1/9 

1/9 

0 

7/27 

IADS1 

2/27 

5/27 

5/27 

10/27 

0 

The  individual  interoperability  measures  are  identical  to  the  ones  calculated  with 
an  instantiated  second  IADS2.  This  supports  the  suggestion  that  the  number  of  systems 
has  no  influence  whatsoever  on  the  interoperability  measurement  as  long  as  their 
individual  character  sets  are  identical.  Should  it  be  found  desirable  to  evaluate  a  certain 
architecture  that  includes  a  system  pair  that  is  identical  in  most  of  its  characters  but  a  few, 
then  these  systems  should  be  included  separately.  The  slightest  difference  in  their 
character  sets  (resulting  out  of  technical,  procedural  or  operational  requirements)  should 
be  thoroughly  analyzed  and  described  by  a  different  character  at  some  place. 
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A5.  Arena  Discrete  Event  Simulation 


This  appendix  provides  more  detail  for  readers  interested  in  the  Arena  discrete 
event  simulation.  There  are  two  pieces  of  the  code  description  for  each  experimental 
component:  a  table  summarizing  various  explanations  and  figures  of  the  actual  code. 


1.  Discrete  event  experimental  component  that  provides  MOEs  not  related  to 
interoperability  measurements  (mission  resource  component ) 


Initialize  Variables  (read  from  file) _ 

■  Arena®  accesses  the  file  “LS  input.xls”,  located  in  the  same  folder  as  the 
simulation  model 


■  The  following  values  are  read  assigned  to  variables  for  later  use  within  the 
simulation  process: 


■  read  day  and  night 
influence  and 
daylight  times: 

simulation  will  ignore 
nighttime  if  value  is  set  to 
‘O’ 

start  of  daylight  [hrs] 
end  of  daylight  [hrs] 

v_day_night(l,l) 

v_day_night(l,2) 

v_day_night(l,3) 

■  read  weather 
influence: 

simulation  will  ignore  bad 
weather  if  value  is  set  to 
‘O’;  if  set  to  value  other 
than  ‘0’  bad  weather  will 
prohibit  use  of  E/O  and  IR 
sensors 

v_weather 

■  read  random  event: 

currently  disabled  (might 
be  used  to  generate  an 
event  later  in  the 
simulation) 

v_random_event 

■  read  grid 
parameters: 

gridsize  (number  of  rows 
and  columns) 

keypad  size  (side  length  of 
one  keypad  element  in  km) 

v_gridsize_rows 

v_gridsize_columns 

v_keypad_size 

■  read  coverage 
limit: 

desired  maximum  number 
of  systems  assigned  to 
cover  an  event 

v_aircraft_limit 

■  read  distances: 

distance  from  homebase  to 
AOR  entry  point 
distance  from  CAOC  to 
AOR  (to  enforce  BLOS 

v_distances(l,l) 

v_distances(2,l) 
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dependence) 

■  read  system 
names: 

currently  disabled,  since 
string  functions  in  Arena 
could  not  be  utilized 
sufficiently 

v_system_name(x,l) 

■  read  system 
designator: 

system  designator  number 
{ 1,  2,  3,  4,  5};  system 
designator  will  be  used  sub 
sequentially  in  order  to 
address  a  certain  system’s 
attributes  (x) 

v_aircraft_type 

■  read  number  of 
systems: 

total  number  of  systems  of 
a  certain  type 

v  number  of  systems(x, 

1) 

■  read  number  of 
systems  at 
homebase: 

number  of  systems  that 
will  be  stationed  at 
homebase  at  start  of 
simulation  run 

v  number  of  systems(x, 

2) 

■  read  system 
speeds: 

cruise  speed  of  a  system 
(not  loiter  speed);  required 
to  determine  transition 
times  between  homebase 
and  AOR) 

v_speed(x,l) 

■  read  maximum 
mission  times 

overall  mission  time  of  a 
specific  system 

v_loiter_time(x,  1 ) 

■  read  E/O  sensor 
capability 

a  given  system’s  E/O 
capability 

v_sensor_type(x,  1 ) 

■  read  IR  sensor 
capability 

a  given  system’s  IR 
capability 

v_sensor_type(x,2) 

■  read  SAR  sensor 
capability 

a  given  system’s  SAR 
capability 

v_sensor_type(x,3) 

■  read  sensor 
footprint  radius 

sensor  footprint  (radius,  in 
km)  on  the  ground;  if  two 
or  more  sensors  are 

available  for  a  certain 
system,  the  maximum 
footprint  should  be 
provided 

v_sensor_footprint(x,  1 ) 

■  read  SATCOM 
capability  (BLOS) 

v_satcom_capable(x,l) 

-  read  VOICE 
communications 
capability  (LOS) 

v_voice_capable(x,  1 ) 

■  read  aircraft 
reliability 

reliability  value  for  the 
aircraft 

v_reliability(x,l) 

■  read  sensor 

reliability  value  for  the 

v_reliability(x,2) 
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reliability 

sensor 

■  read 

reliability  value  for  the 

v_reliability(x,3) 

communications 

communications 

equipment 

reliability 

equipment 

■  read  nighttime 

system’s  capability  to 

v_night_ops(x,l) 

operations 

operate  at  nighttime 

capability 

(sensor  and/or  aircraft) 

Create  Event  (location,  dav/nieht,  weather) 

■  The  simulation  will  initially  determine  an  event  that  happens  at  a  random  hour 
(full  hour  between  start  of  daylight  and  one  hour  before  daylight  ends);  this 
approach  was  chosen  to  allow  for  reaction  time  to  reassign  E/O  systems  to  cover 
the  event.  The  random  event  location  (keypad  row  and  column)  will  be  selected  as 
well. 

■  Chances  for  bad  weather  influence  will  be  specified  using  the  user  input  as 
percentage  value. 

■  The  initially  generated  systems  are  called  to  ‘fill’  the  AOR  (see  below). 

■  Depending  on  the  current  hour  (daytime  or  nighttime)  and  weather  condition,  the 
actual  sensor  requirements  to  cover  an  event  are  determined. 

■  The  sequence  elapses  full  hours  to  set  the  mission  clock. 


Create  Systems  &  Assign  Attributes _ 

■  Initial  generation  of  systems  (up  to  five)  and  respective  numbers  as  specified  by 
user  on  input  sheet  (total  #  of  systems) 

■  Assigning  system  attributes 

■  Split  of  systems:  station  user  defined  number  of  systems  on  homebase,  the  others 
are  deployed  into  AOR  (held  in  ‘AOR  Airborne  Aircraft’  queue  until  called  by 
‘Create  Event’  logic) 

■  Mission  attributes  will  be  assigned  to  AOR  aircraft  (decrease  of  a_time_to joker; 
resembles  that  aircraft  have  already  been  on  station  for  a  while  before  event 
happens) 

■  Aircraft  will  be  assigned  to  event  unless  limit  has  been  reached 

o  Aircraft  that  are  already  within  sensor  reach  of  the  event  will  start 
coverage  of  the  event  immediately 

o  Aircraft  that  are  out  of  reach  of  the  event  will  be  reassigned  to  the  event 
(closest  aircraft  first);  travel  time  to  transition  from  current  location  to 
event  location  will  elapse  before  event  coverage  begins 


Actions  at  Homebase _ 

■  Aircraft  are  assigned  to  a  specific  holding  spot  (shelter)  at  homebase  according  to 
their  system  designator  number 

■  Returning  aircraft  (out  of  AOR)  will  require  turn-around  time  before  they  become 
available  for  further  assignments;  aircraft  that  returned  to  homebase  due  to  a 
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failure  will  require  additional  turn-around  time  for  repairs 

■  Aircraft  launches  can  happen  due  to  a  specific  CAOC  decision  or  as  a  regular 
replacement 

o  Regular  replacement: 

■  mission  parameters  (assigned  to  event  or  regular  mission)  are 
provided 

■  in  case  system  was  not  night  ops  capable  and  would  not  reach 
mission  location  before  nightfall,  launch  will  be  aborted  and 
CAOC  receives  notification  to  reconsider  decision 

o  CAOC  decision: 

■  mission  parameters  (assigned  to  event  or  regular  mission)  are 
provided 

■  Aircraft  transitions  from  homebase  to  AOR  entry  point 


Actions  within  Area  of  Responsibility  (APR) _ 

■  Aircraft  that  reach  AOR  entry  point  (and  initial  deployments  at  run  start)  are 
handled  according  to  their  assignment  status  (assigned  to  event  or  regular 
mission) 

■  Aircraft  transition  between  ‘stations’  according  to  travel  time: 

o  transition  from  entry  point  to  mission  area  and  back 
o  reassignment  from  mission  area  to  event  location  at  run  start 

■  Travel  times  are  deducted  from  available  mission  time  (time  to  joker  =  time  at 
which  a  regular  replacement  needs  to  be  called  in  to  allow  for  seamless  transfer  of 
duties;  time  to  joker  is  determined  as  20%  of  overall  mission  time) 

■  Before  entering  the  loiter  ‘stations’  (over  event  or  regular  mission)  the  simulation 
checks  for  possible  system  failures  (aircraft,  sensor  or  communications 
equipment);  if  random  failure  gate  trips,  the  failure  code  is  recorded  and  time  to 
joker  reduced  by  random  (understand  as:  the  system  knows  beforehand  whether 
or  not  it  will  malfunction) 

■  After  joker  time  is  reached,  the  procedure  will  determine  the  further  progress 
before  continuing  mission  until  bingo  time: 

o  No  failure: 

A  regular  replacement  will  be  called  in  (same  system  type)  if  available;  if 
not  available,  CAOC  decision  is  required 
o  Communications  failure: 

system  will  continue  its  mission  (gather  data  for  later  analysis)  until 
reaching  bingo  time;  replacement  will  be  handled  by  CAOC  decision 
(CAOC  needs  time  to  realize  comms  outage;  see  below) 
o  Aircraft  or  sensor  failure: 

system  will  abort  its  mission  immediately;  CAOC  decision  is  required  to 
handle  replacement 

■  When  bingo  time  is  reached  (or  after  immediate  mission  abort  due  to  failure)  the 
aircraft  transitions  to  AOR  entry  point  and  from  there  to  homebase  for  turn¬ 
around 
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CAOC  Decision  Actions _ 

■  CAOC  decisions  are  required  in  all  unforeseen  events  (launch  abort,  mission  abort 
after  failure,  re-tasking  if  equivalent  system  is  not  available  at  homebase): 
o  After  called  upon,  time  for  decision  process  starts.  Time  for  situation 
analysis  (and  realization  of  a  comms  failure)  will  be  added 
o  First  choice  is  to  replace  an  specific  aircraft  with  the  same  type,  if 
available 

o  In  case  no  systems  are  available  at  any  homebase,  time  will  be  added  for 
reconsideration  before  decision  cycle  starts  again 
o  Decision  maker  calls  for  another  aircraft  based  on  a  availability  (higher 
number  of  aircraft  at  a  specific  homebase  increases  its  chances  to  be  called 
upon) 

o  Selected  aircraft  is  checked  for  night  ops  or  sufficient  time  before  night 
ops  would  be  required;  if  system  does  not  fulfill  night  ops  requirement 
then  CAOC  decision  cycle  will  be  reinitiated  after  a  certain  amount  of 
reconsideration  time 

o  A  successful  decision  process  calls  the  appropriate  system  off  its 
homebase  and  mission  orders  are  transmitted 
o  CAOC  decision  time  is  recorded  in  the  end 


Display  Area _ 

■  Several  approaches  to  display  viable  information  on  the  run  progress  is  provided: 
o  Theater  overview: 

■  shows  aircraft  locations  (at  homebase,  in  transition,  covering 
event,  performing  regular  mission) 

■  shows  level  of  mission  time  left  (to  reach  joker  time  or  bingo  time) 

■  provides  overall  information  on  runtime,  daytime,  weather  and 
number  of  aircraft 

o  Plot  areas: 

■  Several  observations  are  plotted  over  time: 

•  number  of  aircraft  assigned  to  event 

•  number  of  aircraft  covering  event 

•  number  of  aircraft  in  AOR 

•  number  of  failures  (aircraft,  sensor,  comms,  overall) 

•  covered  area  (overlapping  coverage  areas  are  NOT 
accounted  for) 
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2.  Discrete  event  experimental  component  that  provides  the  MOE  related  to 
interoperability  measurements  (system  interoperability  component ) 


Initialize  Variables  (read  from  file) _ 

■  Arena®  accesses  the  file  “JCA  applied.xls”,  located  in  the  same  folder  as  the 
simulation  model 


■  The  following  values  are  read  assigned  to  variables  for  later  use  within  the 
simulation  process:  


■  read  Battlespace 
Awareness  data 
transformation 
(receive/transmit)  10 
character: 

“0”  or  “1”  defines  10 
character.  BA  refers 
to  a  separate  input 
“Row”  in  the 
simulation 

B  A_Transform_Tx 

B  A_Transorm_Rx 

■  read  Battlespace 

“0”  or  “1”  defines  10 

BA_Categorization_Tx 

Awareness 
categorization 
(receive/transmit)  10 
character: 

character.  BA  refers 
to  a  separate  input 
“Row”  in  the 
simulation 

B  A_Categorization_Rx 

■  read  Battlespace 

“0”  or  “1”  defines  10 

B  A_Collection_E/0_Tx 

Awareness 

characters  for  either 

B  A_Collection_E/ 0_Rx 

collection 

IR/E/O/SAR  sensors. 

B  A_Collection_IR_T  x 

(receive/transmit)  10 

BA  refers  to  a 

B  A_Collection_IR_Rx 

characters: 

separate  input  “Row” 
in  the  simulation 

B  A_Collection_S  AR_Rx 

B  A Collection S  AR T  x 

■  read  Battlespace 

“0”  or  “1”  defines  10 

BA_Analysis_Tx 

Awareness 
analysis/production 
(receive/transmit)  10 
characters: 

character.  BA  refers 
to  a  separate  input 
“Row”  in  the 
simulation 

B  A_Analy  sis_Rx 

■  read  Net-centric 

0”  or  “1”  defines  10 

NC_LOS_Tx 

LOS 

(receive/transmit)  10 
characters: 

character.  NC  refers 
to  a  separate  input 
“Row”  in  the 
simulation 

NC_LOS_Rx 

■  read  Net-centric 

0”  or  “1”  defines  10 

NC_LOS_Tx 

BLOS 

(receive/transmit)  10 
characters: 

character.  NC  refers 
to  a  separate  input 
“Row”  in  the 
simulation 

NC_LOS_Rx 
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■  read  Command  and 
Control 

(receive/transmit  10 
characters: 

0”  or  “1”  defines  10 
character.  CC  refers 
to  a  separate  input 
“Row”  in  the 
simulation 

CC_Issue_Orders_Rx 

CC_Issue_Orders_Tx 

Results 

■  The  Interoperability  character  simulation  recorded  two  statistics:  Record 


Number  (number  of  paths  for  each  entity)  and  the  Record  Time  (total  time  for 
each  entity).  Instead  of  utilizing  the  software’s  statistical  models  available,  the 
“Tools  >  ReportDatabase>Export  Summary  Statistics  to  CSV  (comma 
separated  values)  file”  function  was  utilized  to  export  the  data  to  a  Microsoft® 
Excel®  spreadsheet.  These  results  are  available  in  Chapter  3  and  4. 
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A6.  Complete  Collection  of  System  Interoperability  Experimental  Results 
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Experiment  IE- 3  ( add  BLOS  capability  to  ARGUS-IS) 
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